When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
ALiBi can silently zero out attention weights through floating-point underflow.
The paper says ALiBi’s linear bias scaling can make affected attention heads partly blind, hurting token retrieval more than standard decoder benchmarks reveal. The authors report the issue in state-of-the-art pretrained ALiBi models and test it with 148M-parameter decoder pretraining runs. They evaluate four training-time mitigations, with log-scaled distances giving the most consistent passkey-retrieval gains. Default ALiBi slopes still remain strong in some retrieval settings, especially needle-in-a-haystack tests. HF Daily Papers' note
The paper says ALiBi’s linear bias scaling can make affected attention heads partly blind, hurting token retrieval more than standard decoder benchmarks reveal. The authors report the issue in state-of-the-art pretrained ALiBi models and test it with 148M-parameter decoder pretraining runs. They evaluate four training-time mitigations, with log-scaled distances giving the most consistent passkey-retrieval gains. Default ALiBi slopes still remain strong in some retrieval settings, especially needle-in-a-haystack tests. HF Daily Papers' note
score 5