Attend at the Right Resolution: Head-Adaptive Sparse Attention via Query-Key Interaction Structure
HASA reports up to 14.07× faster attention than dense attention, without additional training.
The method adapts how it selects context to the query-key interaction structure of each pretrained attention head. It uses a compact, head-specific feature space and merges neighboring token regions when their responses align. Selected regions then receive exact attention using the original full-dimensional keys and values. The authors report capturing 88.1% of dense-attention mass, with accuracy gains over UNIQUE of up to 3.47 points on RULER and 2.57 points on LongBench-Pro.
ArXiv · AI/CL/LG's note
The method adapts how it selects context to the query-key interaction structure of each pretrained attention head. It uses a compact, head-specific feature space and merges neighboring token regions when their responses align. Selected regions then receive exact attention using the original full-dimensional keys and values. The authors report capturing 88.1% of dense-attention mass, with accuracy gains over UNIQUE of up to 3.47 points on RULER and 2.57 points on LongBench-Pro.
ArXiv · AI/CL/LG's note
score 5