Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD trains the student from replayed teacher prefixes instead of running fresh tool-using rollouts.
The paper targets the cost of multi-turn on-policy distillation for agents, where each update normally requires new environment interactions and teacher queries. Its method, Replayed-Prefix On-Policy Distillation, lets the student act at selected points inside pre-collected teacher trajectories while the teacher supplies dense supervision. The authors describe a “prefix trap”: more student-written history can be more relevant but also push the teacher into less reliable states. In their reported tests on Python-based math reasoning and search environments, ReOPD matches or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4x faster per rollout. HF Daily Papers' note
The paper targets the cost of multi-turn on-policy distillation for agents, where each update normally requires new environment interactions and teacher queries. Its method, Replayed-Prefix On-Policy Distillation, lets the student act at selected points inside pre-collected teacher trajectories while the teacher supplies dense supervision. The authors describe a “prefix trap”: more student-written history can be more relevant but also push the teacher into less reliable states. In their reported tests on Python-based math reasoning and search environments, ReOPD matches or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4x faster per rollout. HF Daily Papers' note
score 5