Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

Mid-training distillation can trade away factual recall even as reasoning improves.

The paper says standard forward KL distillation behaves differently once a model is in mid-training on curated corpora. In that phase, post-trained teachers keep boosting reasoning but slow the student’s acquisition of factual recall. The authors attribute the split to teacher confidence varying by domain and to students learning low-entropy factual knowledge earlier. Their proposed Switch Distillation routes confident teacher predictions through distillation and falls back to cross-entropy otherwise, narrowing the factual-recall gap while keeping reasoning gains. HF Daily Papers' note