Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout
Muir reports lower perplexity than Gated DeltaNet-2 at matching throughput, with stable perplexity on streams up to 10 million tokens.
The design assigns fewer parameters to erasure gates, reweights memory rows during readout, and puts the savings into a wider feed-forward network. Across models with 166 million to 1.34 billion parameters, the authors report perplexity reductions of 1.8–2.0% against GDN-2 and 2.5–3.6% against Transformer. The 1.34-billion-parameter model maintains near-100% passkey retrieval through 1 million tokens. A new parallel update stabilizes training at large chunk sizes; Muir reaches GDN-2’s 100-billion-token validation loss 1.2× faster.
HF Daily Papers' note
The design assigns fewer parameters to erasure gates, reweights memory rows during readout, and puts the savings into a wider feed-forward network. Across models with 166 million to 1.34 billion parameters, the authors report perplexity reductions of 1.8–2.0% against GDN-2 and 2.5–3.6% against Transformer. The 1.34-billion-parameter model maintains near-100% passkey retrieval through 1 million tokens. A new parallel update stabilizes training at large chunk sizes; Muir reaches GDN-2’s 100-billion-token validation loss 1.2× faster.
HF Daily Papers' note
score 5