Register Tokens for Bounded-State Reasoning in Diffusion Language Models
Fixed register tokens let diffusion language models carry reasoning state after generated text is cleared.
The paper tests this on LLaDA and Dream by training them to decode a chunk, erase it, and continue from the prompt plus the preserved register states. In the reported comparisons, registers beat carrying forward discrete text on every benchmark, with gains up to 8.5 points on math and 19.5 on code. The authors say the effect is strongest in bounded code generation, where correct outputs often need several chunks. They also report that reinforcement learning can further refine the registers for longer-horizon reasoning tasks. HF Daily Papers' note
The paper tests this on LLaDA and Dream by training them to decode a chunk, erase it, and continue from the prompt plus the preserved register states. In the reported comparisons, registers beat carrying forward discrete text on every benchmark, with gains up to 8.5 points on math and 19.5 on code. The authors say the effect is strongest in bounded code generation, where correct outputs often need several chunks. They also report that reinforcement learning can further refine the registers for longer-horizon reasoning tasks. HF Daily Papers' note
score 5