AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
AURORA-LM keeps text in a decodable continuous latent space and makes the diffusion model handle that harder representation directly.
The paper separates the latent text representation from the model that learns its distribution, using a query-based encoder-decoder and a block-causal diffusion transformer. It generates blocks left to right while denoising positions inside each block in parallel. The authors say it leads evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. A 1B-parameter version trained with about 1500 EFLOPs improves further and beats a larger public latent-diffusion language model under their matched evaluation. HF Daily Papers' note
The paper separates the latent text representation from the model that learns its distribution, using a query-based encoder-decoder and a block-causal diffusion transformer. It generates blocks left to right while denoising positions inside each block in parallel. The authors say it leads evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. A 1B-parameter version trained with about 1500 EFLOPs improves further and beats a larger public latent-diffusion language model under their matched evaluation. HF Daily Papers' note
score 6