Factored One-sided Adam-Moment for Practical and Scalable SOAP
FOAM matches SOAP’s validation loss with half its optimizer-state memory in the authors’ reported tests.
The optimizer drops SOAP’s output-side preconditioning factor, removing one eigendecomposition, and replaces its dense second-moment estimate with row-by-column statistics. The authors report validation loss matching strong SOAP and Muon baselines in LLM pre-training runs from 124 million to 760 million parameters. FOAM also runs at Muon-class wall-clock time and remains robust across a threefold learning-rate range. The exactness claims for its simplifications depend on stated curvature assumptions; the experiments report little effect from deviations.
HF Daily Papers' note
The optimizer drops SOAP’s output-side preconditioning factor, removing one eigendecomposition, and replaces its dense second-moment estimate with row-by-column statistics. The authors report validation loss matching strong SOAP and Muon baselines in LLM pre-training runs from 124 million to 760 million parameters. FOAM also runs at Muon-class wall-clock time and remains robust across a threefold learning-rate range. The exactness claims for its simplifications depend on stated curvature assumptions; the experiments report little effect from deviations.
HF Daily Papers' note
score 5