Denoising Hierarchical Representations: Joint Continuous Diffusion for Language Modeling
The paper claims hierarchical joint diffusion gives continuous diffusion language models a sizable benchmark lift with little added overhead.
Ollu and Komodakis introduce H-CDLMs, which diffuse token-level representations alongside coarser clusters from pretrained token embeddings. Applied to CoBit as H-CoBit, the method reports GenPPL of 49.4 on LM1B and 50.4 on OWT, beating the baseline by 24.2 and 20.7 points. It also reaches 27.4% accuracy on GSM8K, above prior continuous diffusion and flow-based models cited in the abstract. The same framework is applied to FLM as H-FLM, where the authors report consistent gains. ArXiv · AI/CL/LG's note
Ollu and Komodakis introduce H-CDLMs, which diffuse token-level representations alongside coarser clusters from pretrained token embeddings. Applied to CoBit as H-CoBit, the method reports GenPPL of 49.4 on LM1B and 50.4 on OWT, beating the baseline by 24.2 and 20.7 points. It also reaches 27.4% accuracy on GSM8K, above prior continuous diffusion and flow-based models cited in the abstract. The same framework is applied to FLM as H-FLM, where the authors report consistent gains. ArXiv · AI/CL/LG's note
score 5