Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
An intermediate teacher checkpoint helped select what to transfer, yielding the highest average scores among compared distillation methods in all six tested settings.
Waypoint-OPD checks whether a teacher’s early and late reinforcement-learning updates push action preferences in the same direction. It suppresses the training signal where those shifts oppose each other, adding one frozen-teacher scoring pass over endpoint-only transfer. Tests used Qwen3-4B teachers and three larger Qwen3 students on mathematical and code reasoning. Removing or shuffling the gates reduced mathematical reasoning performance in experiments with the 8B student.
HF Daily Papers' note
Waypoint-OPD checks whether a teacher’s early and late reinforcement-learning updates push action preferences in the same direction. It suppresses the training signal where those shifts oppose each other, adding one frozen-teacher scoring pass over endpoint-only transfer. Tests used Qwen3-4B teachers and three larger Qwen3 students on mathematical and code reasoning. Removing or shuffling the gates reduced mathematical reasoning performance in experiments with the 8B student.
HF Daily Papers' note
score 4