Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
OPD helps a student model explore better reasoning paths, but only when the teacher signal is reliable.
The paper frames on-policy distillation as dense token-level guidance, not a way to raise the student’s underlying capability ceiling. It identifies two failure modes: teacher-student mismatch, where guidance points away from correctness, and length exploitation, where the objective rewards truncated or padded answers. The authors test lightweight fixes, including advantage clipping and log-scale compression, across seven benchmarks. Those regulations reduce the length pathology and outperform OPD variants and RLVR baselines in their experiments. HF Daily Papers' note
The paper frames on-policy distillation as dense token-level guidance, not a way to raise the student’s underlying capability ceiling. It identifies two failure modes: teacher-student mismatch, where guidance points away from correctness, and length exploitation, where the objective rewards truncated or padded answers. The authors test lightweight fixes, including advantage clipping and log-scale compression, across seven benchmarks. Those regulations reduce the length pathology and outperform OPD variants and RLVR baselines in their experiments. HF Daily Papers' note
score 5