Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Teachers can recognize a student model’s mistakes yet still follow them into a wrong answer.
On-policy distillation trains students on their own generated solutions using token-level guidance from a teacher. The paper’s CriticOPD method has the teacher critique an incorrect solution, then lets the student revise it. The teacher scores the revision with access to the critique, while the student trains on the revision without it. Across two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD outperformed standard on-policy distillation, with the largest gains on competition problems.
ArXiv · AI/CL/LG's note
On-policy distillation trains students on their own generated solutions using token-level guidance from a teacher. The paper’s CriticOPD method has the teacher critique an incorrect solution, then lets the student revise it. The teacher scores the revision with access to the critique, while the student trains on the revision without it. Across two Qwen3 students and six mathematical reasoning benchmarks, CriticOPD outperformed standard on-policy distillation, with the largest gains on competition problems.
ArXiv · AI/CL/LG's note
score 5