GUI-Primitives: Diagnosing Spatial Reasoning Failures in Vision-Language GUI Grounding
The benchmark finds GUI agents mostly miss the candidate regions before they ever prove spatial understanding.
GUI-Primitives tests 19 vision-language models on 994 contrastive GUI grounding items across seven spatial relations. The best strict point-in-box accuracy is 32%, with predictions landing outside both designated candidates on 60-92% of items. When predictions do hit a candidate, models do better on left/right, above/below, proximity, and list ordinal, but not on containment or occlusion. Marking the two candidates boosts selection accuracy by 35-57 points, presented as a diagnostic rather than a deployable fix. HF Daily Papers' note
GUI-Primitives tests 19 vision-language models on 994 contrastive GUI grounding items across seven spatial relations. The best strict point-in-box accuracy is 32%, with predictions landing outside both designated candidates on 60-92% of items. When predictions do hit a candidate, models do better on left/right, above/below, proximity, and list ordinal, but not on containment or occlusion. Marking the two candidates boosts selection accuracy by 35-57 points, presented as a diagnostic rather than a deployable fix. HF Daily Papers' note
score 5