Multilinguality in Hybrid Attention LLMs
Hybrid attention models may handle multilingual representations better when full attention comes first.
The paper studies how mixing recurrent-style attention with full attention affects multilingual behavior in LLMs. Its interpretability work finds cross-lingual alignment spikes around the first full-attention layer, suggesting layer order matters. In multilingual distillation tests, alternative orderings beat the standard setup throughout training, with learning up to 2.5x faster. HF Daily Papers' note
The paper studies how mixing recurrent-style attention with full attention affects multilingual behavior in LLMs. Its interpretability work finds cross-lingual alignment spikes around the first full-attention layer, suggesting layer order matters. In multilingual distillation tests, alternative orderings beat the standard setup throughout training, with learning up to 2.5x faster. HF Daily Papers' note
score 4