MUON-RT Reusing the Teacher's Trajectory for Momentum-Free Muon for LLM Self-Distillation
Muon-RT reports higher accuracy with 27% less persistent training memory than momentum Muon in a Qwen2.5-7B self-distillation test.
The method uses the gap between an exponential-moving-average teacher and its student to replace momentum buffers for hidden weight matrices. In Self-Distillation Fine-Tuning on Qwen2.5-7B-Instruct, the authors report average accuracy gains of 5.4 points over momentum Muon and 2.6 points over AdamW, with 42% less persistent memory than AdamW. A separate policy-optimization experiment on Olmo-3-7B-Instruct reports a 2.0-point accuracy gain over AdamW. The paper’s theoretical convergence result depends on explicit regularity and alignment assumptions.
HF Daily Papers' note
The method uses the gap between an exponential-moving-average teacher and its student to replace momentum buffers for hidden weight matrices. In Self-Distillation Fine-Tuning on Qwen2.5-7B-Instruct, the authors report average accuracy gains of 5.4 points over momentum Muon and 2.6 points over AdamW, with 42% less persistent memory than AdamW. A separate policy-optimization experiment on Olmo-3-7B-Instruct reports a 2.0-point accuracy gain over AdamW. The paper’s theoretical convergence result depends on explicit regularity and alignment assumptions.
HF Daily Papers' note
score 4