Expert-Space Exploration in MoE Reinforcement Learning
ESRL treats MoE routing itself as the exploration space for RL post-training.
The paper argues that perturbing expert selection can increase rollout diversity, but naive perturbation can send tokens through unsuitable experts and hurt quality. ESRL keeps high-confidence experts fixed, samples only from plausible alternatives, and adjusts perturbation strength based on router entropy. It also records the expert paths used during rollout and replays them during optimization to reduce routing mismatch. On Qwen3-30B-A3B, the authors report gains over GRPO of 3.2 points on average Pass@1 and 4.5 points on Pass@8. HF Daily Papers' note
The paper argues that perturbing expert selection can increase rollout diversity, but naive perturbation can send tokens through unsuitable experts and hurt quality. ESRL keeps high-confidence experts fixed, samples only from plausible alternatives, and adjusts perturbation strength based on router entropy. It also records the expert paths used during rollout and replays them during optimization to reduce routing mismatch. On Qwen3-30B-A3B, the authors report gains over GRPO of 3.2 points on average Pass@1 and 4.5 points on Pass@8. HF Daily Papers' note
score 4