Semantic Head Specialization Guides Hybrid ViT Attention for Multimodal LLMs
The paper argues that ViT attention heads split into object and background specialists, and that this split predicts multimodal performance.
The authors call the pattern Semantic Head Specialization and propose SHS-Index to measure it. They say full attention shows the strongest specialization, while chunk-window designs differ in measurable ways. Using those findings, they build Ariadne Attention, which they report matches full attention across 22 image and video tasks with 6.5x less attention compute. ArXiv · AI/CL/LG's note
The authors call the pattern Semantic Head Specialization and propose SHS-Index to measure it. They say full attention shows the strongest specialization, while chunk-window designs differ in measurable ways. Using those findings, they build Ariadne Attention, which they report matches full attention across 22 image and video tasks with 6.5x less attention compute. ArXiv · AI/CL/LG's note
score 5