Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling
The paper argues that frozen transformers can act as generative samplers using only examples in the prompt.
The authors prove constructions where attention computes responsibility weights and empirical averages, while feedforward layers carry out Euler-style sampler updates. They cover closed-form diffusion, smoothed diffusion, and an approximation to energy-based sampling. In experiments on semantic-topic prompts, hidden states show a two-stage pattern: toward a uniform spherical reference, then back to topic-specific structure near the output. ArXiv · AI/CL/LG's note
The authors prove constructions where attention computes responsibility weights and empirical averages, while feedforward layers carry out Euler-style sampler updates. They cover closed-form diffusion, smoothed diffusion, and an approximation to energy-based sampling. In experiments on semantic-topic prompts, hidden states show a two-stage pattern: toward a uniform spherical reference, then back to topic-specific structure near the output. ArXiv · AI/CL/LG's note
score 6