AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
The paper keeps text in a high-capacity continuous latent space and makes the diffusion model handle the harder representation.
AURORA-LM separates the text autoencoder from the model that learns the latent distribution. Its Block-causal Diffusion Transformer generates blocks left to right while denoising positions inside each block in parallel. The authors report top results among evaluated continuous and diffusion-based language models on OpenWebText generation and XSum summarization. A 1B-parameter run with about 1500 EFLOPs improves further and beats a larger public latent-diffusion language model under matched evaluation. ArXiv · AI/CL/LG's note
AURORA-LM separates the text autoencoder from the model that learns the latent distribution. Its Block-causal Diffusion Transformer generates blocks left to right while denoising positions inside each block in parallel. The authors report top results among evaluated continuous and diffusion-based language models on OpenWebText generation and XSum summarization. A 1B-parameter run with about 1500 EFLOPs improves further and beats a larger public latent-diffusion language model under matched evaluation. ArXiv · AI/CL/LG's note
score 5