Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
The paper claims exact long-context MoE training can be made memory-bounded by capping four separate peak allocations at once.
The authors target expert dispatch, vocabulary projection, checkpoint boundaries, and optimizer state, arguing that any one of them can break training as context, batch size, model size, or device count changes. Their schedules change computation and data movement order, not the resulting loss or gradients. In component tests, they report peak cuts of up to 59.3% for MoE dispatch and 86.6% for vocabulary projection, plus a 2.05x faster offloaded optimizer step. Combined on 120B to 667B parameter MoE models, the method trains at 1M context length with 8x to 32x the reach of a tuned FSDP2 baseline and up to 10.4x its throughput. HF Daily Papers' note
The authors target expert dispatch, vocabulary projection, checkpoint boundaries, and optimizer state, arguing that any one of them can break training as context, batch size, model size, or device count changes. Their schedules change computation and data movement order, not the resulting loss or gradients. In component tests, they report peak cuts of up to 59.3% for MoE dispatch and 86.6% for vocabulary projection, plus a 2.05x faster offloaded optimizer step. Combined on 120B to 667B parameter MoE models, the method trains at 1M context length with 8x to 32x the reach of a tuned FSDP2 baseline and up to 10.4x its throughput. HF Daily Papers' note
score 5