Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Layer dropout cut training compute by up to 25% and later enabled up to 1.5x faster inference in the authors’ LLM experiments.
The paper argues that layer dropout should return to modern LLM pre-training recipes when its distribution, schedule, and optimizer settings are tuned. Across more than 2,400 experiments, the authors report lower loss at the same training FLOPs, or similar-to-lower validation loss for the same step count with less compute. They also found post-training gains from early exit, intermediate-layer skipping, and self-speculative decoding, with negligible accuracy loss. The study spans models from 271M to 8.2B parameters and datasets up to 160B tokens. HF Daily Papers' note
The paper argues that layer dropout should return to modern LLM pre-training recipes when its distribution, schedule, and optimizer settings are tuned. Across more than 2,400 experiments, the authors report lower loss at the same training FLOPs, or similar-to-lower validation loss for the same step count with less compute. They also found post-training gains from early exit, intermediate-layer skipping, and self-speculative decoding, with negligible accuracy loss. The study spans models from 271M to 8.2B parameters and datasets up to 160B tokens. HF Daily Papers' note
score 5