What Does Privileged Information Add to On-Policy Self-Distillation?
Privileged references added only modestly beyond self-distillation in the paper’s main Qwen tests.
The authors built AMPLE-Math, a 5,319-problem math suite with six reasoning views sharing the same answer, to compare privileged teacher inputs against matched reference-free distillation. In Qwen3-1.7B, much of the gain came from reference-free distillation under thinking-enabled evaluation, with the clearest extra benefit coming from a polished solution. SmolLM3-3B showed a larger trace benefit at one checkpoint, adding two percentage points at step 50. Longer thinking-enabled rollouts reversed gains into losses in both model families under otherwise fixed conditions. Source: ArXiv · AI/CL/LG's note
The authors built AMPLE-Math, a 5,319-problem math suite with six reasoning views sharing the same answer, to compare privileged teacher inputs against matched reference-free distillation. In Qwen3-1.7B, much of the gain came from reference-free distillation under thinking-enabled evaluation, with the clearest extra benefit coming from a polished solution. SmolLM3-3B showed a larger trace benefit at one checkpoint, adding two percentage points at step 50. Longer thinking-enabled rollouts reversed gains into losses in both model families under otherwise fixed conditions. Source: ArXiv · AI/CL/LG's note
score 4