Towards demystifying the creativity of diffusion models
Google argues diffusion models make new outputs because training smooths the score function, pushing samples into interpolation zones instead of exact training points.
The note ties that “creativity” to regularization, including weight decay and implicit effects from gradient-based training. In a simple 1-D example, smoothing softens the sharp boundary that would otherwise send every sample back to one of two memorized points. In higher-dimensional data, Google says the same effect helps samples reach the data manifold while reducing collapse onto specific training examples. The work was presented at ICLR 2026, with code released for the numerical experiments. Google Research's note
The note ties that “creativity” to regularization, including weight decay and implicit effects from gradient-based training. In a simple 1-D example, smoothing softens the sharp boundary that would otherwise send every sample back to one of two memorized points. In higher-dimensional data, Google says the same effect helps samples reach the data manifold while reducing collapse onto specific training examples. The work was presented at ICLR 2026, with code released for the numerical experiments. Google Research's note
score 5