RangeSWA: Training-Free Length Extrapolation for Continuous Diffusion Language Models
The authors report less repetition and lower generative perplexity when diffusion models produce text beyond their training length, without additional training.
RangeSWA combines sliding-window attention with summaries of distant text blocks and dynamic content retrieval. It also adjusts positional mapping, conditioning attention, and window sizes across denoising stages to limit generation degradation. Experiments covered multiple continuous diffusion language models. The abstract gives no numerical results or tested generation lengths.
ArXiv · AI/CL/LG's note
RangeSWA combines sliding-window attention with summaries of distant text blocks and dynamic content retrieval. It also adjusts positional mapping, conditioning attention, and window sizes across denoising stages to limit generation degradation. Experiments covered multiple continuous diffusion language models. The abstract gives no numerical results or tested generation lengths.
ArXiv · AI/CL/LG's note
score 5