Training Vision Transformers to Focus: Emergence of Spatial Attention Head Specialization
Two training losses help vision transformers develop attention heads that focus on distinct image regions without annotations.
The researchers target self-supervised models such as DINO, whose representations can be ambiguous or overly similar. Their losses steer individual attention heads toward specific regions, encouraging more spatially selective and diverse attention. Qualitative and quantitative evaluations report improved interpretability and performance on dense prediction tasks, including zero-shot image segmentation. The abstract does not give numerical gains.
HF Daily Papers' note
The researchers target self-supervised models such as DINO, whose representations can be ambiguous or overly similar. Their losses steer individual attention heads toward specific regions, encouraging more spatially selective and diverse attention. Qualitative and quantitative evaluations report improved interpretability and performance on dense prediction tasks, including zero-shot image segmentation. The abstract does not give numerical gains.
HF Daily Papers' note
score 4