On-Demand Attention: Language Models Know When to Recall
The paper says a model can predict when full context will help before it spends compute reading that context.
The authors introduce On-Demand Attention, a local-first decoding method that adds a small recall head while leaving pretrained weights unchanged. When the recall head predicts benefit, the model invokes global attention over the stored KV cache. Their vLLM implementation uses GPU-side conditional execution to turn fewer global reads into long-context decoding speedups. Tests on Qwen and Gemma models show it recovers much of the quality lost with local attention while cutting global attention use. ArXiv · AI/CL/LG's note
The authors introduce On-Demand Attention, a local-first decoding method that adds a small recall head while leaving pretrained weights unchanged. When the recall head predicts benefit, the model invokes global attention over the stored KV cache. Their vLLM implementation uses GPU-side conditional execution to turn fewer global reads into long-context decoding speedups. Tests on Qwen and Gemma models show it recovers much of the quality lost with local attention while cutting global attention use. ArXiv · AI/CL/LG's note
score 6