MMOE: Modernizing Diffusion Transformers with Efficient Expert Design
The paper argues sparse-expert diffusion transformers need LLM-style efficiency design, not just more parameters.
ModernMOE adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual reuse to SiT-style diffusion transformers. The authors say all experiments used one eight-H100 node, batch size 256, for 400k steps. Under matched protocols, MMOE reached lower FID at every recorded checkpoint than dense and intermediate sparse-expert baselines. Routing analysis showed stable expert specialization, heavy use of lightweight routes, and limited denoising-step route churn. ArXiv · AI/CL/LG's note
ModernMOE adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual reuse to SiT-style diffusion transformers. The authors say all experiments used one eight-H100 node, batch size 256, for 400k steps. Under matched protocols, MMOE reached lower FID at every recorded checkpoint than dense and intermediate sparse-expert baselines. Routing analysis showed stable expert specialization, heavy use of lightweight routes, and limited denoising-step route churn. ArXiv · AI/CL/LG's note
score 5