Loopholing All the Way: Language Modeling with pure Self-Conditioning
LAW solved 42.2% of GSM8K problems, compared with 34.5% for uniform Loopholing Discrete Diffusion.
The model passes its predicted expected embeddings between steps and decodes tokens only at the end, making self-conditioning the core generation method. The reported GSM8K comparison follows training on TinyGSM, with sampling at temperature 1 and 1,024 model evaluations. Fine-tuning the pretrained Duo diffusion model with LAW raised its accuracy from 17.2% to 48.1%. The authors also report stronger Sudoku and Maze results and performance approaching autoregressive models on OpenWebText at 64 evaluations.
ArXiv · AI/CL/LG's note
The model passes its predicted expected embeddings between steps and decodes tokens only at the end, making self-conditioning the core generation method. The reported GSM8K comparison follows training on TinyGSM, with sampling at temperature 1 and 1,024 model evaluations. Fine-tuning the pretrained Duo diffusion model with LAW raised its accuracy from 17.2% to 48.1%. The authors also report stronger Sudoku and Maze results and performance approaching autoregressive models on OpenWebText at 64 evaluations.
ArXiv · AI/CL/LG's note
score 5