Training Specialist Models without Reasoning Trajectories for Domain Expert Distillation
Tuning a specialist model changes the hidden reasoning it passes to students, even when no gold reasoning traces are used.
The paper treats student distillation as a probe for the teacher’s latent trajectory choices. Across 27 specialist-student pairings, the authors report a strong match between the specialist’s specialization/generalization profile and the distilled student’s profile. By controlling distributional drift, they shift both teacher and student along a trade-off between domain precision and retention of general capability. The experiments cover chemistry, physics, and multilingual settings, including cases where teacher and student come from different model families. HF Daily Papers' note
The paper treats student distillation as a probe for the teacher’s latent trajectory choices. Across 27 specialist-student pairings, the authors report a strong match between the specialist’s specialization/generalization profile and the distilled student’s profile. By controlling distributional drift, they shift both teacher and student along a trade-off between domain precision and retention of general capability. The experiments cover chemistry, physics, and multilingual settings, including cases where teacher and student come from different model families. HF Daily Papers' note
score 4