GTFormer: A Syntax-Free Graph Network Alternative to Transformers for 8192$\times$ Length Extrapolation in Linear Time
Trained on sequences of at most 512 tokens, GTFormer reportedly maintained near-perfect accuracy at more than 4 million tokens.
The paper reports results across eight code, log, and multi-hop retrieval benchmarks, with no additional training for longer sequences. GTFormer replaces self-attention with a hierarchical graph that combines a binary tree, cross-boundary connections, random shortcuts, and a top-down broadcast pass. The reported tests ran on a single commodity GPU with memory use scaling linearly with sequence length. Standard and RoPE Transformers suffered attention collapse or exhausted memory in those comparisons.
ArXiv · AI/CL/LG's note
The paper reports results across eight code, log, and multi-hop retrieval benchmarks, with no additional training for longer sequences. GTFormer replaces self-attention with a hierarchical graph that combines a binary tree, cross-boundary connections, random shortcuts, and a top-down broadcast pass. The reported tests ran on a single commodity GPU with memory use scaling linearly with sequence length. Standard and RoPE Transformers suffered attention collapse or exhausted memory in those comparisons.
ArXiv · AI/CL/LG's note
score 5