Sliding-window beats linear attention
The paper says sliding-window attention with sinks matches or beats post-trained linear attention, without extra training.
The authors tested this across multiple LLMs and downstream tasks. On Needle-in-a-Haystack and BABILong, they report SWA scoring 2 to 10 times higher than linear attention. They argue SWA is fast, low-memory, and cheap enough to be the default retrofit for cutting inference memory. HF Daily Papers' note
The authors tested this across multiple LLMs and downstream tasks. On Needle-in-a-Haystack and BABILong, they report SWA scoring 2 to 10 times higher than linear attention. They argue SWA is fast, low-memory, and cheap enough to be the default retrofit for cutting inference memory. HF Daily Papers' note
score 6