Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
The paper says low-bit reasoning improves when the student is trained on its own quantized trajectories, not fixed teacher-forced prefixes.
The authors argue that sub-3-bit quantization worsens exposure bias during long autoregressive reasoning, causing loops and unfinished solutions. Their on-policy distillation stage lets the quantized student generate through the same forward path used at deployment, then gets supervision from a frozen full-precision teacher on those prefixes. Across four models at 2.79 and 1.88 effective bits, it raised reported BF16 performance retention to 70% on MATH-500 and 91% on HumanEval. The gains beat continued teacher-forced QAD under matched budgets while preserving short-form performance. ArXiv · AI/CL/LG's note
The authors argue that sub-3-bit quantization worsens exposure bias during long autoregressive reasoning, causing loops and unfinished solutions. Their on-policy distillation stage lets the quantized student generate through the same forward path used at deployment, then gets supervision from a frozen full-precision teacher on those prefixes. Across four models at 2.79 and 1.88 effective bits, it raised reported BF16 performance retention to 70% on MATH-500 and 91% on HumanEval. The gains beat continued teacher-forced QAD under matched budgets while preserving short-form performance. ArXiv · AI/CL/LG's note
score 5