When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better
A simple offline rollout recipe beat on-policy distillation in 14 of 17 tested teacher-student pairs.
The paper tests Semi-OPD, which distills from rollouts generated by the initial student rather than continually sampling on-policy during training. Across model pairs from 1.5B to 235B parameters, it reports gains of up to 13.6% accuracy and an 11.4x training speedup over OPD. The authors say OPD helps mainly when teacher and student outputs are already closely aligned, measured by output-token overlap. For less aligned pairs, longer student rollouts drift away from the teacher and weaken the distillation signal. HF Daily Papers' note
The paper tests Semi-OPD, which distills from rollouts generated by the initial student rather than continually sampling on-policy during training. Across model pairs from 1.5B to 235B parameters, it reports gains of up to 13.6% accuracy and an 11.4x training speedup over OPD. The authors say OPD helps mainly when teacher and student outputs are already closely aligned, measured by output-token overlap. For less aligned pairs, longer student rollouts drift away from the teacher and weaken the distillation signal. HF Daily Papers' note
score 5