Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference
Layer dropout cut training compute by up to 25% and supported faster inference without meaningful accuracy loss.
The paper argues that layer dropout belongs back in modern LLM pre-training when its distribution, schedule, and optimizer settings are tuned. At equal training FLOPs, the authors report lower loss; at a fixed step count, they report similar or lower validation loss while using less compute. They also say the same setup enables early exit, intermediate-layer skipping, and self-speculative decoding, reaching up to 1.5x inference speedup with negligible accuracy loss. The study covers more than 2,400 training runs, from 271M to 8.2B parameters and up to 160B tokens. ArXiv · AI/CL/LG's note
The paper argues that layer dropout belongs back in modern LLM pre-training when its distribution, schedule, and optimizer settings are tuned. At equal training FLOPs, the authors report lower loss; at a fixed step count, they report similar or lower validation loss while using less compute. They also say the same setup enables early exit, intermediate-layer skipping, and self-speculative decoding, reaching up to 1.5x inference speedup with negligible accuracy loss. The study covers more than 2,400 training runs, from 271M to 8.2B parameters and up to 160B tokens. ArXiv · AI/CL/LG's note
score 6