On the Design Space of Continuous Diffusion Language Models
The paper says two training levers, self-conditioning and embedding geometry, explain much of what makes continuous diffusion language models work.
The authors argue self-conditioning helps by reducing a mismatch between training states and inference states. They introduce DoubleCat to recover those gains without an extra forward pass, and SemiCat to address token identifiability with discrete reveals. SemiCat reaches 49.94% on TinyGSM with standard training hyperparameters and a tuned sampler, and at 1.7B beats discrete diffusion on four of six likelihood-based evaluations. The paper also finds cross-entropy pushes embeddings toward a geometry with decoding error close to one-hot representations. ArXiv · AI/CL/LG's note
The authors argue self-conditioning helps by reducing a mismatch between training states and inference states. They introduce DoubleCat to recover those gains without an extra forward pass, and SemiCat to address token identifiability with discrete reveals. SemiCat reaches 49.94% on TinyGSM with standard training hyperparameters and a tuned sampler, and at 1.7B beats discrete diffusion on four of six likelihood-based evaluations. The paper also finds cross-entropy pushes embeddings toward a geometry with decoding error close to one-hot representations. ArXiv · AI/CL/LG's note
score 5