Graph Machine: Towards Better Pretraining via Edges
Graph Machine keeps a large linear-size state while routing through only a tiny number of tokens per sparse layer.
The paper proposes “edges” as differentiable, pointer-like objects for dynamic sparse access. Its test replaces 75% of dense Transformer layers in Qwen3-0.6B and pretrains from scratch on 15.7B tokens. Retrieving 2 of 4,096 tokens per KV head only slightly worsened loss; retrieving 4 marginally improved the best model. ArXiv · AI/CL/LG's note
The paper proposes “edges” as differentiable, pointer-like objects for dynamic sparse access. Its test replaces 75% of dense Transformer layers in Qwen3-0.6B and pretrains from scratch on 15.7B tokens. Retrieving 2 of 4,096 tokens per KV head only slightly worsened loss; retrieving 4 marginally improved the best model. ArXiv · AI/CL/LG's note
score 6