Looped Uniform Diffusion Language Models
Reusing early Transformer layers cut training compute by up to 3.3× while matching a non-looped model’s validation loss.
The researchers repeatedly apply a few early layers within each denoising step, leaving the training objective and sampler unchanged. Matching a baseline trained on about 0.5 trillion tokens required up to 2.8× fewer training FLOPs on OpenWebText and 3.3× fewer on FineWeb-Edu. At a matched function-evaluation budget, generation perplexity fell by up to 9.8%, with most gains at moderate loop counts. Favoring smaller loop counts during training also reduced downstream inference loops by 12% without degrading accuracy.
ArXiv · AI/CL/LG's note
The researchers repeatedly apply a few early layers within each denoising step, leaving the training objective and sampler unchanged. Matching a baseline trained on about 0.5 trillion tokens required up to 2.8× fewer training FLOPs on OpenWebText and 3.3× fewer on FineWeb-Edu. At a matched function-evaluation budget, generation perplexity fell by up to 9.8%, with most gains at moderate loop counts. Favoring smaller loop counts during training also reduced downstream inference loops by 12% without degrading accuracy.
ArXiv · AI/CL/LG's note
score 4