Sonar-MoE: In-Backward Expert Updates for Memory-Efficient Mixture-of-Experts Training
Sonar-MoE cut peak GPU memory by 19%–22% in fine-tuning tests on two MoE models.
It updates expert tensors during the backward pass and immediately frees their gradients, keeping at most one layer’s expert gradients in GPU memory. Tests on OLMoE-1B-7B and Granite-3.1-3B-A800M found steps took 17%–23% less time than standard training with global-norm clipping. Updates were bitwise identical to standard training without expert gradient clipping, and held-out loss stayed within 0.003 of all baselines. Updating whole batched tensors avoided the 20%–72% slowdown seen with individual expert updates at the same memory usage.
ArXiv · AI/CL/LG's note
It updates expert tensors during the backward pass and immediately frees their gradients, keeping at most one layer’s expert gradients in GPU memory. Tests on OLMoE-1B-7B and Granite-3.1-3B-A800M found steps took 17%–23% less time than standard training with global-norm clipping. Updates were bitwise identical to standard training without expert gradient clipping, and held-out loss stayed within 0.003 of all baselines. Updating whole batched tensors avoided the 20%–72% slowdown seen with individual expert updates at the same memory usage.
ArXiv · AI/CL/LG's note
score 5