Abstract: Denoising Diffusion Probabilistic Models (DDPMs) are a very popular class of deep generative model that have been successfully applied to a diverse range of problems including image and video generation, protein and material synthesis, weather forecasting, and neural surrogates of partial differential equations. Despite their ubiquity it is hard to find an introduction to DDPMs which is simple, comprehensive, clean and clear. The compact explanations necessary in research papers are not able to elucidate all of the different design steps taken to formulate the DDPM and the rationale of the steps that are presented is often omitted to save space. Moreover, the expositions are typically presented from the variational lower bound perspective which is unnecessary and arguably harmful as it obfuscates why the method is working and suggests generalisations that do not perform well in practice. On the other hand, perspectives that take the continuous time-limit are beautiful and general, but they have a high barrier-to-entry as they require background knowledge of stochastic differential equations and probability flow. In this note, we distill down the formulation of the DDPM into six simple steps each of which comes with a clear rationale. We assume that the reader is familiar with fundamental topics in machine learning including basic probabilistic modelling, Gaussian distributions, maximum likelihood estimation, and deep learning.
Introduction: Demystifying Diffusion
Denoising Diffusion Probabilistic Models (DDPMs) have become a cornerstone of modern generative AI. However, the mathematical foundations are often presented through the lens of complex Variational Lower Bounds (ELBO) or advanced Stochastic Differential Equations (SDEs), creating a significant barrier to entry. This note provides a "middle way," reframing DDPMs as a sequence of simple supervised regression problems. By breaking the formulation into six logical design steps, we clarify the "why" behind the math using standard machine learning concepts.
The Six-Step Framework
1. Augmentation
The first step converts the difficult unsupervised task of modeling a complex data distribution into a series of simpler supervised regression problems. This is achieved by adding $T$ levels of noise to the data, creating a sequence of increasingly "low-fidelity" versions of the original input.
2. Supervised Step-wise Objective
For each noise level, we define a maximum-likelihood objective. The goal is to learn a model $p(x^{(t-1)}|x^{(t)})$ that can reverse the noising process, effectively "denoising" the data one step at a time.
3. Parameter Tying & Joint Objective
Training $T$ separate models is inefficient. Instead, we use a single neural network that takes the noise level $t$ as an additional input (amortization). This allows the model to share information across different noise levels and scales the approach to large $T$. The joint objective becomes a weighted sum of the individual step-wise objectives.
4. Gaussian Regression Model
By choosing a Gaussian model for the denoising process, we can perform one of the averages in the training objective analytically. This "Rao-Blackwellization" trick effectively trains the model on an infinite set of noise augmentations, significantly reducing variance in the gradients.
5. Parameterization
We must decide what the neural network actually predicts: the clean image $x^{(0)}$, the noise $\epsilon$, or the score function. While mathematically related, these choices lead to different practical behaviors and connect DDPMs to denoising auto-encoders and score matching.
6. Schedule Selection
The final step is selecting the "noise schedule"—how the variance of the added noise changes across the $T$ steps. In the limit of large $T$, the specific schedule matters less than the Signal-to-Noise Ratio (SNR) at the boundaries of the process.
Key Insights and Unified Theory
Reframing diffusion reveals it as a neural auto-regressive model applied to an augmented dataset. Instead of predicting pixels, it predicts cleaner versions of the entire image. This perspective critiques the standard ELBO approach, which often suggests learning the forward noising process—a generalization that typically performs poorly compared to fixed linear Gaussian noising.
Furthermore, this six-step skeleton provides a "Rosetta Stone" for the field. Popular variants like the simplified DDPM objective (Ho et al., 2020) or Variational Diffusion Models (Kingma et al., 2021) can be seen as specific configurations of these six design choices.
Conclusion
By stripping away unnecessary mathematical complexity, we see that DDPMs are built on intuitive principles of data augmentation and supervised regression. This pedagogical framework not only makes diffusion more accessible but also clarifies the design space for future research in generative modeling.
Summary updated