Sliding-window beats linear attention
A cheaper attention baseline matched or beat retrofitted linear-attention models across the paper’s tests.
The authors compare sliding-window attention with sinks against post-trained linear-attention LLMs and find SWA performs as well or better on multiple downstream tasks. On long-context reasoning benchmarks, including Needle-in-a-Haystack and BABILong, SWA is reported at 2x to 10x higher performance. The paper says SWA needs no post-training, runs fast, and uses low memory, making it the authors’ recommended inference-memory shortcut over retrofitting linear attention. ArXiv · AI/CL/LG's note
The authors compare sliding-window attention with sinks against post-trained linear-attention LLMs and find SWA performs as well or better on multiple downstream tasks. On long-context reasoning benchmarks, including Needle-in-a-Haystack and BABILong, SWA is reported at 2x to 10x higher performance. The paper says SWA needs no post-training, runs fast, and uses low memory, making it the authors’ recommended inference-memory shortcut over retrofitting linear attention. ArXiv · AI/CL/LG's note
score 6