Distributionally Robust Mixture-of-Experts Training
The paper argues MoE reliability should be trained against bad routing outcomes, not just balanced expert traffic.
DRMoET treats layer-wise experts as robustness groups and upweights high-loss routing paths with an entropy-regularized softmax over smoothed expert losses. In FLAME-MoE runs at 746M and 10.3B total parameters, it beat standard FLAME-MoE and an auxiliary-loss-free balancing baseline. At 10.3B parameters and 67B tokens, the seven-task average rose from 0.6625 to 0.6767. The authors also report lower expert-loss variance, reduced excess loss under forced mid-k misrouting, and stronger domain-expert specialization. HF Daily Papers' note
DRMoET treats layer-wise experts as robustness groups and upweights high-loss routing paths with an entropy-regularized softmax over smoothed expert losses. In FLAME-MoE runs at 746M and 10.3B total parameters, it beat standard FLAME-MoE and an auxiliary-loss-free balancing baseline. At 10.3B parameters and 67B tokens, the seven-task average rose from 0.6625 to 0.6767. The authors also report lower expert-loss variance, reduced excess loss under forced mid-k misrouting, and stronger domain-expert specialization. HF Daily Papers' note
score 5