Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
SkewAdam cuts optimizer state for a 6.78B-parameter MoE model from 50.6 GB to 1.29 GB while fitting peak training memory under 40 GB.
The paper assigns different optimizer state to the dense backbone, experts, and router instead of treating all MoE parameters alike. In its controlled run over 82M tokens, SkewAdam reports validation perplexity of 108.4, ahead of AdamW, Muon, and Lion. The author says the tiering mainly buys memory, not accuracy, since an ablation with far more state reaches the same perplexity. HF Daily Papers' note
The paper assigns different optimizer state to the dense backbone, experts, and router instead of treating all MoE parameters alike. In its controlled run over 82M tokens, SkewAdam reports validation perplexity of 108.4, ahead of AdamW, Muon, and Lion. The author says the tiering mainly buys memory, not accuracy, since an ablation with far more state reaches the same perplexity. HF Daily Papers' note
score 5