Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining
The paper proposes an optimizer add-on meant to speed Muon by pushing harder along flat curvature directions during LLM pretraining.
The method combines slow- and fast-decay momentum only in those flat directions, aiming to reduce noise while still adapting quickly to curvature. A sphere constraint is used to avoid parameter inflation and the effective learning-rate decay the authors say a naive version would cause. They report acceleration across dense and MoE models from 0.12B to 2.3B parameters, and include theory for why the flat-direction multiscale momentum should help. Source: ArXiv · AI/CL/LG's note.
The method combines slow- and fast-decay momentum only in those flat directions, aiming to reduce noise while still adapting quickly to curvature. A sphere constraint is used to avoid parameter inflation and the effective learning-rate decay the authors say a naive version would cause. They report acceleration across dense and MoE models from 0.12B to 2.3B parameters, and include theory for why the flat-direction multiscale momentum should help. Source: ArXiv · AI/CL/LG's note.
score 4