When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors
The paper says VLMs can encode spatial relations locally, then lose them in the representation used downstream.
Garg and Shani introduce EICS and CASP-VL to test whether VLM representations keep concepts stable under controlled spatial changes. Across multiple architectures, spatial relations were represented less faithfully than lexical information, despite above-chance benchmark performance. The relation could be found at the predicate tokens, but disappeared when aggregated into EOS or mean-pooled sentence representations. ArXiv · AI/CL/LG's note
Garg and Shani introduce EICS and CASP-VL to test whether VLM representations keep concepts stable under controlled spatial changes. Across multiple architectures, spatial relations were represented less faithfully than lexical information, despite above-chance benchmark performance. The relation could be found at the predicate tokens, but disappeared when aggregated into EOS or mean-pooled sentence representations. ArXiv · AI/CL/LG's note
score 5