Pretraining Data Scale Reverses the Muon-AdamW Fine-Tuning Gap
Larger models need fewer pretraining tokens per parameter before AdamW pulls ahead after fine-tuning.
Muon reduces pretraining loss faster, but that advantage does not consistently survive fine-tuning in the authors’ study. They introduce a margin-to-spread ratio measuring how far the correct output score sits above incorrect scores relative to their spread. The ratio favors Muon at smaller data budgets and AdamW at larger ones, matching the fine-tuning results. A simplified linear associative-memory model provides a one-update proof of a related effect, with simulations showing it persists through longer training.
ArXiv · AI/CL/LG's note
Muon reduces pretraining loss faster, but that advantage does not consistently survive fine-tuning in the authors’ study. They introduce a margin-to-spread ratio measuring how far the correct output score sits above incorrect scores relative to their spread. The ratio favors Muon at smaller data budgets and AdamW at larger ones, matching the fine-tuning results. A simplified linear associative-memory model provides a one-update proof of a related effect, with simulations showing it persists through longer training.
ArXiv · AI/CL/LG's note
score 5