LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
The paper says MoE diffusion language models scale differently enough from autoregressive models to need their own training rules.
The authors find that optimal batch size rises faster and learning rate falls faster as compute grows. Their IsoFLOP analysis points to a slight preference for more data over more activated model compute. Using those findings, they trained LLaDA MoE v2, a 30B-A3B diffusion model, from scratch on 23.5T tokens. They report that it nears Qwen3 on several knowledge, reasoning, and coding benchmarks despite using about 65% as many pretraining tokens, and beats SDAR Chat on seven of eight reasoning and coding tests after supervised fine-tuning alone. HF Daily Papers' note
The authors find that optimal batch size rises faster and learning rate falls faster as compute grows. Their IsoFLOP analysis points to a slight preference for more data over more activated model compute. Using those findings, they trained LLaDA MoE v2, a 30B-A3B diffusion model, from scratch on 23.5T tokens. They report that it nears Qwen3 on several knowledge, reasoning, and coding benchmarks despite using about 65% as many pretraining tokens, and beats SDAR Chat on seven of eight reasoning and coding tests after supervised fine-tuning alone. HF Daily Papers' note
score 5