MuonGS: Pretraining LLMs with Relaxed Weight-Norm Control
MuonGS reports an estimated token-equivalent speedup of up to about 15% over MuonH at matched validation loss in dense-model pretraining.
The optimizer uses Gram–Schmidt projection to remove the radial component of each Muon update without fixing the weight norm. Under bounded updates and a constant nominal learning rate, the authors prove that weight norms grow at most proportionally to the square root of training steps, sustaining effective learning-rate accumulation. Tests on 160M- and 400M-parameter dense models and a mixture-of-experts model with 135M active parameters found lower validation perplexity than MuonH over long training runs.
ArXiv · AI/CL/LG's note
The optimizer uses Gram–Schmidt projection to remove the radial component of each Muon update without fixing the weight norm. Under bounded updates and a constant nominal learning rate, the authors prove that weight norms grow at most proportionally to the square root of training steps, sustaining effective learning-rate accumulation. Tests on 160M- and 400M-parameter dense models and a mixture-of-experts model with 135M active parameters found lower validation perplexity than MuonH over long training runs.
ArXiv · AI/CL/LG's note
score 5