Ambient LLMs: Data-Efficient Supervised Fine-tuning via Temperature Matching
The authors report up to 21% average relative improvement over standard supervised fine-tuning.
Ambient SFT uses a frozen pre-trained model as a noisy teacher to help limit overfitting and forgetting when fine-tuning data is scarce. The fine-tuned model must match the teacher’s conditional predictions after both distributions are softened with temperature. That setting controls trust in the teacher, connecting knowledge distillation at T=1 to ordinary SFT as temperature approaches infinity. The authors report gains across domain adaptation, instruction-tuning and reasoning; the paper is marked work in progress.
ArXiv · AI/CL/LG's note
Ambient SFT uses a frozen pre-trained model as a noisy teacher to help limit overfitting and forgetting when fine-tuning data is scarce. The fine-tuned model must match the teacher’s conditional predictions after both distributions are softened with temperature. That setting controls trust in the teacher, connecting knowledge distillation at T=1 to ordinary SFT as temperature approaches infinity. The authors report gains across domain adaptation, instruction-tuning and reasoning; the paper is marked work in progress.
ArXiv · AI/CL/LG's note
score 5