First-Token Attraction in Mamba Dynamics
Later tokens converge toward the first token’s limiting direction as Mamba models deepen.
The authors prove explicit exponential convergence rates and describe how this attraction produces token clustering and attention-sink effects. Their analytical framework handles matrix-valued interactions that lack the positivity properties used to study attention models. Experiments on Falcon-Mamba-7B show concentration on the first token increasing with depth and dominating in later layers. The analysis also extends to broader sequence dynamics, including state-space models, causal attention and classical time-series models.
ArXiv · AI/CL/LG's note
The authors prove explicit exponential convergence rates and describe how this attraction produces token clustering and attention-sink effects. Their analytical framework handles matrix-valued interactions that lack the positivity properties used to study attention models. Experiments on Falcon-Mamba-7B show concentration on the first token increasing with depth and dominating in later layers. The analysis also extends to broader sequence dynamics, including state-space models, causal attention and classical time-series models.
ArXiv · AI/CL/LG's note
score 4