P^2-MOPD: Accelerating Multi-Teacher On-Policy Distillation via Parameter Priors
The 1.7B model reached 94.7% average recovery, compared with 80.9% for standard multi-teacher on-policy distillation.
P²-MOPD jointly trains the student’s weights and the coefficients used to mix its parameters with those of expert models. Keeping the student trainable lets the experts guide learning without restricting the result to combinations of frozen models. Across four domains at 1.7B and 4B scales, the authors report faster, more balanced recovery from the same initialization. The method surpassed MOPD’s final average recovery in half the training steps.
HF Daily Papers' note
P²-MOPD jointly trains the student’s weights and the coefficients used to mix its parameters with those of expert models. Keeping the student trainable lets the experts guide learning without restricting the result to combinations of frozen models. Across four domains at 1.7B and 4B scales, the authors report faster, more balanced recovery from the same initialization. The method surpassed MOPD’s final average recovery in half the training steps.
HF Daily Papers' note
score 4