Megadose The AI curve, compressed.

Delayed Effects of Temporary Weight Decay in Muon Pretraining

· ArXiv · AI/CL/LG ·
A decay schedule that initially hurt a 1.4B-parameter model ultimately beat both no decay and the best sampled constant-decay baseline.

Selected at 162M parameters and transferred without retuning, the schedule left validation loss 0.26 nats worse than no decay when removed. It pulled ahead only after 73% of training and finished 0.028 nats better. Experiments at 162M and 405M parameters indicate that smaller weight norms after decay explain most of the final improvement, keeping relative update steps larger. The authors conclude that temporary interventions should be judged at the intended final training budget, not just when they end.

ArXiv · AI/CL/LG's note

score 5

Categories: Research