SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking
SAS trains the sparse-attention selector against prediction loss instead of copying dense attention rankings.
The paper says hard Top-K selection in existing trainable sparse attention blocks gradients, pushing those methods toward layer-wise distillation of dense attention distributions. SAS injects continuous selector scores into attention logits during training so standard backpropagation can tune context ranking under a fixed attention budget. The authors point to log-form gates inside softmax, normalized gates for historical context, and preserved continuous scores as important implementation choices. They also report a memory-efficient Triton kernel and gains over trainable sparse-attention baselines on reasoning, long-context, and agentic tasks, especially with tight budgets. HF Daily Papers' note
The paper says hard Top-K selection in existing trainable sparse attention blocks gradients, pushing those methods toward layer-wise distillation of dense attention distributions. SAS injects continuous selector scores into attention logits during training so standard backpropagation can tune context ranking under a fixed attention budget. The authors point to log-form gates inside softmax, normalized gates for historical context, and preserved continuous scores as important implementation choices. They also report a memory-efficient Triton kernel and gains over trainable sparse-attention baselines on reasoning, long-context, and agentic tasks, especially with tight budgets. HF Daily Papers' note
score 5