Two mathematical models of knowledge distillation
The paper’s models attribute distillation’s gains to faster optimization, rather than better generalization.
For generalized linear models trained with stochastic gradient descent, the authors prove that distillation reaches performant student models asymptotically faster than training without it. For rank-1 matrix approximation, they identify conditions on the target matrix under which distillation makes gradient descent converge strictly faster than the supervised objective. Experiments on real datasets corroborate the theoretical predictions. The findings establish an optimization benefit within the settings analyzed.
ArXiv · AI/CL/LG's note
For generalized linear models trained with stochastic gradient descent, the authors prove that distillation reaches performant student models asymptotically faster than training without it. For rank-1 matrix approximation, they identify conditions on the target matrix under which distillation makes gradient descent converge strictly faster than the supervised objective. Experiments on real datasets corroborate the theoretical predictions. The findings establish an optimization benefit within the settings analyzed.
ArXiv · AI/CL/LG's note
score 5