Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation
Standard multi-teacher distillation recovered only 35.6% of the oracle ensemble’s available headroom; Open-MOPD raised that to 83.4%.
The paper sets up a controlled benchmark on SmolLM3-3B-Base with oracle routing to isolate capability integration. It finds the shortfall comes from token-level optimization budget imbalance, not gradient conflict. The authors identify sequence-length disparities, uneven convergence rates, and stale rewards from asynchronous updates as separate causes. Open-MOPD addresses them with token-share balancing, gap-aware budget allocation, and student reward refresh. HF Daily Papers' note
The paper sets up a controlled benchmark on SmolLM3-3B-Base with oracle routing to isolate capability integration. It finds the shortfall comes from token-level optimization budget imbalance, not gradient conflict. The authors identify sequence-length disparities, uneven convergence rates, and stale rewards from asynchronous updates as separate causes. Open-MOPD addresses them with token-share balancing, gap-aware budget allocation, and student reward refresh. HF Daily Papers' note
score 4