On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Qwen3.8-Flash-Next claims near-predecessor benchmark performance with far less active compute.
The paper describes a 125B-parameter sparse MoE model that activates 6B parameters per token, plus 51B off-accelerator n-gram embedding parameters. It beats the 397B-A17B predecessor on eight of fourteen pre-training benchmarks and trails by no more than 2.6 points on the others. The authors attribute the efficiency gains to a hybrid GDN/global-attention stack, Qwen Sparse Attention during continued pretraining, gated residual branches, and host-prefetched n-gram tables. They also report that Muon and the architecture raise optimal learning rate and batch size while improving training stability. HF Daily Papers' note
The paper describes a 125B-parameter sparse MoE model that activates 6B parameters per token, plus 51B off-accelerator n-gram embedding parameters. It beats the 397B-A17B predecessor on eight of fourteen pre-training benchmarks and trails by no more than 2.6 points on the others. The authors attribute the efficiency gains to a hybrid GDN/global-attention stack, Qwen Sparse Attention during continued pretraining, gated residual branches, and host-prefetched n-gram tables. They also report that Muon and the architecture raise optimal learning rate and batch size while improving training stability. HF Daily Papers' note
score 7