ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation
The paper argues pruned LLMs often still know how to generate useful answers, but their best completions get buried behind repetition.
ShortOPD trains the compressed model on its own on-policy states while using the original model as a frozen teacher. The method shortens early rollouts by detecting teacher-confirmed repetitive suffixes, then expands the budget as the policy can use longer generations. Across math, code, and open-ended tasks, the authors report about a 9x recovery over the unrecovered compressed model and 1.6x-4.4x over standard recovery recipes. It reached within two points of an 8192-token rollout setup using 8.5 hours instead of 35.9 and 71% fewer rollout tokens. HF Daily Papers' note
ShortOPD trains the compressed model on its own on-policy states while using the original model as a frozen teacher. The method shortens early rollouts by detecting teacher-confirmed repetitive suffixes, then expands the budget as the policy can use longer generations. Across math, code, and open-ended tasks, the authors report about a 9x recovery over the unrecovered compressed model and 1.6x-4.4x over standard recovery recipes. It reached within two points of an 8192-token rollout setup using 8.5 hours instead of 35.9 and 71% fewer rollout tokens. HF Daily Papers' note
score 4