Eliciting Weak-to-Strong Generalization with On-Policy Reverse Distillation
The paper proposes a training method meant to let a stronger student use a weaker teacher without inheriting the teacher’s ceiling.
On-Policy Reverse Distillation evaluates how the teacher’s policy shifts on the student’s own rollouts, then amplifies verifier-supported updates in that direction. The authors say this keeps the same stationary points as policy optimization while speeding learning past the teacher. In their tests, OPRD improves performance with fewer student updates across successive transfer and multi-teacher settings. HF Daily Papers' note
On-Policy Reverse Distillation evaluates how the teacher’s policy shifts on the student’s own rollouts, then amplifies verifier-supported updates in that direction. The authors say this keeps the same stationary points as policy optimization while speeding learning past the teacher. In their tests, OPRD improves performance with fewer student updates across successive transfer and multi-teacher settings. HF Daily Papers' note
score 5