It's Not RoPE that Creates Sinks: The Role of Self-Concentration and Value-Non-Mixing in Attention
The paper argues that attention sinks can arise from causal-mask-driven attention patterns, not RoPE itself.
The authors connect initial-position Attention Sink behavior with Massive Activations that can complicate low-bit quantization. Their experiments point to self-concentration of attention and later value-non-mixing in attention outputs as contributing factors. The effect appears regardless of which token sits at the initial position. The paper was accepted at EMNLP 2026. ArXiv · AI/CL/LG's note
The authors connect initial-position Attention Sink behavior with Massive Activations that can complicate low-bit quantization. Their experiments point to self-concentration of attention and later value-non-mixing in attention outputs as contributing factors. The effect appears regardless of which token sits at the initial position. The paper was accepted at EMNLP 2026. ArXiv · AI/CL/LG's note
score 5