Self-Distillation with Verbal Regularization
Prompting the teacher to use privileged answers less forcefully helped self-distillation avoid early overconfidence.
The paper introduces VRSD, which adds system-prompt regularization to on-policy self-distillation so the teacher model rethinks rather than locks onto privileged information. The authors argue this preserves more output entropy and reduces shortcut reasoning. Across three models and four tasks, VRSD beat the OPSD baseline in 11 of 12 settings, with an average Pass@1 gain of 3.9 percentage points. HF Daily Papers' note
The paper introduces VRSD, which adds system-prompt regularization to on-policy self-distillation so the teacher model rethinks rather than locks onto privileged information. The authors argue this preserves more output entropy and reduces shortcut reasoning. Across three models and four tasks, VRSD beat the OPSD baseline in 11 of 12 settings, with an average Pass@1 gain of 3.9 percentage points. HF Daily Papers' note
score 4