Subquadratic Attention with a Hierarchical Tree-Structured Indexer
ZHA cuts attention’s scaling cost—including key selection—while preserving performance in the authors’ evaluations.
The method builds a tree of compressed key summaries, then routes queries through it to select a sparse subset of cached keys. The authors report adapting pretrained models with a short KL-distillation stage over a few billion tokens while maintaining long-context retrieval, reasoning, mathematics and agentic coding performance. Their performance model projects substantial gains over competing methods beyond one million tokens. Context windows reaching hundreds of millions or billions of tokens remain a potential extension, not a demonstrated result.
ArXiv · AI/CL/LG's note
The method builds a tree of compressed key summaries, then routes queries through it to select a sparse subset of cached keys. The authors report adapting pretrained models with a short KL-distillation stage over a few billion tokens while maintaining long-context retrieval, reasoning, mathematics and agentic coding performance. Their performance model projects substantial gains over competing methods beyond one million tokens. Context windows reaching hundreds of millions or billions of tokens remain a potential extension, not a demonstrated result.
ArXiv · AI/CL/LG's note
score 5