On-Policy Delta Distillation
The paper replaces teacher imitation with a “delta signal” meant to isolate what reasoning tuning added.
That signal is defined as the difference between a teacher model and its pre-instruction-tuned base model. The authors argue it gives a more direct reward for transferring reasoning capability during on-policy distillation. In experiments on math, science, and code-reasoning benchmarks, their OPD² method outperformed conventional on-policy distillation and reached strong results with a short post-training period. HF Daily Papers' note
That signal is defined as the difference between a teacher model and its pre-instruction-tuned base model. The authors argue it gives a more direct reward for transferring reasoning capability during on-policy distillation. In experiments on math, science, and code-reasoning benchmarks, their OPD² method outperformed conventional on-policy distillation and reached strong results with a short post-training period. HF Daily Papers' note
score 5