Training Communication-Efficient Mixture-of-Experts Language Models with Layer Re-Configuration
The paper says MoE training can cut communication cost by routing experts through fewer layers instead of every mixing block.
Sun and Waleffe describe CE-MoE, a heterogeneous layout that separates token-mixing depth from channel-mixing expert capacity. The model keeps depth with extra token-mixing and dense-FFN layers, while concentrating routed MoE capacity in selected layers. Across 2B to 31.5B total parameters, the authors report matched validation loss and downstream results against full-MoE baselines under matched parameter budgets. At 31.5B, they report 33.3% fewer GPU-hours, plus better average downstream score and inference throughput. ArXiv · AI/CL/LG's note
Sun and Waleffe describe CE-MoE, a heterogeneous layout that separates token-mixing depth from channel-mixing expert capacity. The model keeps depth with extra token-mixing and dense-FFN layers, while concentrating routed MoE capacity in selected layers. Across 2B to 31.5B total parameters, the authors report matched validation loss and downstream results against full-MoE baselines under matched parameter budgets. At 31.5B, they report 33.3% fewer GPU-hours, plus better average downstream score and inference throughput. ArXiv · AI/CL/LG's note
score 5