On-Policy Self-Distillation with Teacher Evolution
Self-distillation stalls when a model gives itself unreliable guidance, the authors find.
Their proposed TESD method uses feedback on task correctness and context independence to improve the self-teacher during training. The same feedback helps estimate guidance reliability, reducing the weight of samples where the teacher is unreliable. Across multiple datasets and model scales, the authors report that TESD consistently outperforms representative token-level and sequence-level methods.
HF Daily Papers' note
Their proposed TESD method uses feedback on task correctness and context independence to improve the self-teacher during training. The same feedback helps estimate guidance reliability, reducing the weight of samples where the teacher is unreliable. Across multiple datasets and model scales, the authors report that TESD consistently outperforms representative token-level and sequence-level methods.
HF Daily Papers' note
score 5