Multi-Token Prediction Done Right
Soft-MTP reports better accuracy and faster speculative decoding with no added parameters or measurable overhead.
The preprint identifies a noisy component in standard multi-token prediction gradients that interferes with learning and obscures a useful self-distillation signal. Soft-MTP removes that component through a one-line code change, reusing existing cross-entropy computations. Tests on models from 167 million to 1.2 billion parameters show lower next-token perplexity and higher zero-shot accuracy at every scale. At 1.2 billion parameters, it also improves math and code reasoning after fine-tuning.
HF Daily Papers' note
The preprint identifies a noisy component in standard multi-token prediction gradients that interferes with learning and obscures a useful self-distillation signal. Soft-MTP removes that component through a one-line code change, reusing existing cross-entropy computations. Tests on models from 167 million to 1.2 billion parameters show lower next-token perplexity and higher zero-shot accuracy at every scale. At 1.2 billion parameters, it also improves math and code reasoning after fine-tuning.
HF Daily Papers' note
score 5