Looped Uniform Diffusion Language Models
Reusing early Transformer layers cut training compute by up to 3.3× while matching a non-looped model’s validation loss.
The researchers repeatedly apply a few early layers within each denoising step, leaving the training objective and sampler unchanged. Matching a baseline trained on about 0.5 trillion tokens required up to 2.8× fewer training FLOPs on OpenWebText and 3.3× fewer on FineWeb-Edu. At the same number of model evaluations, looped models achieved up to 9.8% lower generation perplexity, with most gains at moderate loop counts. Favoring smaller loop counts during training also reduced downstream inference loops by 12% without reducing accuracy.
HF Daily Papers' note
The researchers repeatedly apply a few early layers within each denoising step, leaving the training objective and sampler unchanged. Matching a baseline trained on about 0.5 trillion tokens required up to 2.8× fewer training FLOPs on OpenWebText and 3.3× fewer on FineWeb-Edu. At the same number of model evaluations, looped models achieved up to 9.8% lower generation perplexity, with most gains at moderate loop counts. Favoring smaller loop counts during training also reduced downstream inference loops by 12% without reducing accuracy.
HF Daily Papers' note
score 4