Rethinking Relative Policy Differences Transfer of On-Policy Distillation
The paper argues that OPD works best when it transfers task-specific policy changes, not raw model strength.
The authors frame relative log-probability differences as an implicit reward for KL-regularized policy optimization. Their experiments find that differences from RL or supervised fine-tuning transfer more reliably than differences caused by scale alone. The expert supplies the behavior to transfer, while the reference model helps strip away shared behavior and isolate what matters. Transfer still depends on whether the induced update moves the target toward the expert and generalizes beyond sampled rollouts. HF Daily Papers' note
The authors frame relative log-probability differences as an implicit reward for KL-regularized policy optimization. Their experiments find that differences from RL or supervised fine-tuning transfer more reliably than differences caused by scale alone. The expert supplies the behavior to transfer, while the reference model helps strip away shared behavior and isolate what matters. Transfer still depends on whether the induced update moves the target toward the expert and generalizes beyond sampled rollouts. HF Daily Papers' note
score 4