Seen but Unsaid: Locating and Repairing Cross-Image Comparison in Vision-Language Models
Updating just 0.02% of model parameters improved cross-image matching accuracy by up to 28.3 percentage points.
The study identifies two failures: models can encode an exact match without reporting it, while semantic matches remain in visual tokens without reaching the answer. Tests across Qwen3-VL, InternVL3.5 and Gemma-3 isolate a small set of attention heads carrying cross-image comparisons. Training only those heads’ query and key projections raised SPair-71k accuracy by 28.3 points in Qwen3-VL-8B and 25.1 points in InternVL3.5-8B, with improvements transferring to HPatches and BLINK.
ArXiv · AI/CL/LG's note
The study identifies two failures: models can encode an exact match without reporting it, while semantic matches remain in visual tokens without reaching the answer. Tests across Qwen3-VL, InternVL3.5 and Gemma-3 isolate a small set of attention heads carrying cross-image comparisons. Training only those heads’ query and key projections raised SPair-71k accuracy by 28.3 points in Qwen3-VL-8B and 25.1 points in InternVL3.5-8B, with improvements transferring to HPatches and BLINK.
ArXiv · AI/CL/LG's note
score 5