Graph Machine: Towards Better Pretraining via Edges
Graph Machine swaps most dense Transformer layers for sparse edge-based routing while keeping access to a large state.
The paper proposes an architecture with an `O(n)` state reached through dynamic, differentiable pointer-like edges. In a Qwen3-0.6B pretraining run on 15.7B tokens, the author replaces 75% of dense Transformer layers with GM sparse layers. Retrieving only 2 of 4,096 tokens per KV head slightly worsens loss; retrieving 4 marginally improves it. Source: HF Daily Papers' note.
The paper proposes an architecture with an `O(n)` state reached through dynamic, differentiable pointer-like edges. In a Qwen3-0.6B pretraining run on 15.7B tokens, the author replaces 75% of dense Transformer layers with GM sparse layers. Retrieving only 2 of 4,096 tokens per KV head slightly worsens loss; retrieving 4 marginally improves it. Source: HF Daily Papers' note.
score 6