When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
ALiBi can silently lose attention weights when its distance bias falls below floating-point precision.
The paper says this underflow can make affected attention heads partly blind, hurting token retrieval more than standard decoder benchmarks reveal. The authors show the failure in state-of-the-art pretrained ALiBi models and run 148M-parameter decoder pretraining experiments to separate it from ordinary out-of-context degradation. They test four training-time mitigations, with log-scaled distances giving the most consistent passkey-retrieval gains. Default ALiBi slopes still remain a strong baseline, especially for needle-in-a-haystack retrieval. ArXiv · AI/CL/LG's note
The paper says this underflow can make affected attention heads partly blind, hurting token retrieval more than standard decoder benchmarks reveal. The authors show the failure in state-of-the-art pretrained ALiBi models and run 148M-parameter decoder pretraining experiments to separate it from ordinary out-of-context degradation. They test four training-time mitigations, with log-scaled distances giving the most consistent passkey-retrieval gains. Default ALiBi slopes still remain a strong baseline, especially for needle-in-a-haystack retrieval. ArXiv · AI/CL/LG's note
score 6