ELSAA: Efficient Low-Rank and Sparse Attention Approximation for Training Transformers
The paper targets the attention matrix itself, splitting it into sparse high-similarity links and a low-rank global summary.
ELSAA keeps the Transformer’s learned projection and output matrices dense, then approximates the attention score operator after Q, K, and V are produced. Its sparse branch preserves selected sharp token interactions, while its low-rank branch compresses broader context. The authors add a denominator-aware fusion term so the two branches are weighted by estimated attention mass rather than naively combined. The stated goal is longer-context Transformer training without materializing the full quadratic score matrix. ArXiv · AI/CL/LG's note
ELSAA keeps the Transformer’s learned projection and output matrices dense, then approximates the attention score operator after Q, K, and V are produced. Its sparse branch preserves selected sharp token interactions, while its low-rank branch compresses broader context. The authors add a denominator-aware fusion term so the two branches are weighted by estimated attention mass rather than naively combined. The stated goal is longer-context Transformer training without materializing the full quadratic score matrix. ArXiv · AI/CL/LG's note
score 5