Weak-to-Strong On-Policy Distillation
The paper claims a stronger student model can improve by distilling from supervision sources that are all weaker than it.
W2S-OPD builds a proxy teacher from the logit difference between two smaller models, using that contrast as a capability direction. The student then learns from this proxy on its own rollouts with per-token reverse KL. The authors test contrasts based on post-RL vs. pre-RL models, larger vs. smaller base models, and correct vs. wrong hints. Across math and code benchmarks, they report gains over standard OPD and cases where the student surpasses the domain teacher. HF Daily Papers' note
W2S-OPD builds a proxy teacher from the logit difference between two smaller models, using that contrast as a capability direction. The student then learns from this proxy on its own rollouts with per-token reverse KL. The authors test contrasts based on post-RL vs. pre-RL models, larger vs. smaller base models, and correct vs. wrong hints. Across math and code benchmarks, they report gains over standard OPD and cases where the student surpasses the domain teacher. HF Daily Papers' note
score 5