How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
FP8 with asynchronous rollouts reached the same distillation quality 1.62× sooner than synchronous BF16 training in the paper’s tests.
On-policy distillation tolerated substantial differences between response generation and training, but quality fell beyond a model-dependent mismatch budget. Controlled comparisons found it stable under rollout settings that destabilized reinforcement learning methods. Model-specific FP8 recipes preserved quality while supporting at least 1.93× as many KV-cache tokens as BF16. The authors introduced a lightweight check to screen rollout configurations against a calibrated budget without running full training for each candidate.
HF Daily Papers' note
On-policy distillation tolerated substantial differences between response generation and training, but quality fell beyond a model-dependent mismatch budget. Controlled comparisons found it stable under rollout settings that destabilized reinforcement learning methods. Model-specific FP8 recipes preserved quality while supporting at least 1.93× as many KV-cache tokens as BF16. The authors introduced a lightweight check to screen rollout configurations against a calibrated budget without running full training for each candidate.
HF Daily Papers' note
score 5