Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Caching only the teacher’s top-K logits let the authors remove the teacher from training memory while keeping loss nearly unchanged.
The paper reports offline distillation running about 29% faster per iteration and up to 41% higher throughput on a single H200 GPU. Its fused, chunked KL loss avoids materializing the full vocabulary-sized logits tensor, cutting the memory spike that limits long-context runs. That let the authors train at 32,768 tokens on one GPU, four times the context length they cite for the baseline setup. They also release the chunked-loss implementation. HF Daily Papers' note
The paper reports offline distillation running about 29% faster per iteration and up to 41% higher throughput on a single H200 GPU. Its fused, chunked KL loss avoids materializing the full vocabulary-sized logits tensor, cutting the memory spike that limits long-context runs. That let the authors train at 32,768 tokens on one GPU, four times the context length they cite for the baseline setup. They also release the chunked-loss implementation. HF Daily Papers' note
score 5