How Local Mixing Encodes Relative Position in Global NoPE Attention
Hybrid NoPE models may recover position through recency bias carried by local mixing layers.
The paper argues that sliding window attention and gated linear attention inject a recency signal into the residual stream. Global attention layers without explicit positional encoding can then select that signal through their logits. The authors say this differs from pure global NoPE models, where position comes only from the causal mask, because the hybrid recency bias can persist over long sequences. They frame the result as a possible route toward position handling that extrapolates indefinitely. ArXiv · AI/CL/LG's note
The paper argues that sliding window attention and gated linear attention inject a recency signal into the residual stream. Global attention layers without explicit positional encoding can then select that signal through their logits. The authors say this differs from pure global NoPE models, where position comes only from the causal mask, because the hybrid recency bias can persist over long sequences. They frame the result as a possible route toward position handling that extrapolates indefinitely. ArXiv · AI/CL/LG's note
score 4