Early generative diffusion models were conceptualized as discrete-time processes. They operated by taking a data point and iteratively adding small amounts of noise over a fixed number of steps (e.g., $T = 1000$), then training a neural network to reverse each step. While highly effective, this discrete formulation obscured a deeper mathematical reality and restricted models to the specific step schedule chosen during training.
A profound shift occurred when researchers realized that if you shrink the step size toward zero and let the number of steps approach infinity, this discrete sequence of noise additions transforms into a smooth, continuous-time trajectory. This trajectory can be precisely modeled using Stochastic Differential Equations (SDEs).
By reframing diffusion through the lens of continuous-time SDEs, the field achieved a "Grand Unified Theory" that merged disparate generative approaches, unlocked faster and more flexible sampling techniques, and extended diffusion models far beyond simple image generation.
The Intuition: From Discrete Steps to Continuous Vector Fields
In a discrete diffusion model, the forward process moves a data point $x_0$ to a noisy state $x_T$ via a sequence of intermediate distributions. In contrast, the continuous-time framework treats this process as a smooth evolution of a probability distribution over time $t \in [0, T]$.
As formalized by Song et al., the forward process that diffuses data into noise can be written as a standard Itô SDE:
$$\mathrm{d}\mathbf{x} = \mathbf{f}(\mathbf{x}, t)\mathrm{d}t + g(t)\mathrm{d}\mathbf{w}$$
To understand why this works, we can break the equation into two intuitive forces acting on the data:
- The Drift Term ($\mathbf{f}(\mathbf{x}, t)\mathrm{d}t$): This represents a deterministic force that guides the data points over time. For example, it might gently pull data points toward the origin to keep their variance bounded.
- The Diffusion Term ($g(t)\mathrm{d}\mathbf{w}$): This represents the random noise injection. Here, $\mathbf{w}$ is a standard Brownian motion (a continuous random walk), and $g(t)$ is a scalar function controlling the intensity of the noise at time $t$.
As time progresses from $0$ to $T$, any complex, multimodal data distribution (like human faces or molecular structures) is gradually smoothed out and pulled into a simple, uninformative prior distribution, typically a standard Gaussian.
Forward Process (Data to Noise):
[ Complex Data Distribution ] ---> (Drift + Random Noise) ---> [ Gaussian Prior ]
Reverse Process (Noise to Data):
[ Complex Data Distribution ] <--- (Drift - Score Function) <--- [ Gaussian Prior ]
Reversing the Flow via the Score Function
Generating new data requires running this process backward: starting from random noise at $t = T$ and rolling back the clock to $t = 0$. However, you cannot simply reverse a random walk directly, because the random fluctuations ($\mathrm{d}\mathbf{w}$) are inherently unpredictable.
The core breakthrough of the SDE framework relies on a remarkable mathematical result from time-reversal theory. As explained by Wojtowytsch and analyzed by Chen et al., the reverse process is also an SDE that runs backward in time:
$$\mathrm{d}\mathbf{x} = \left[\mathbf{f}(\mathbf{x}, t) - g(t)^2 \nabla_\mathbf{x} \log p_t(\mathbf{x})\right]\mathrm{d}t + g(t)\mathrm{d}\bar{\mathbf{w}}$$
where $\bar{\mathbf{w}}$ is a backward Brownian motion. Notice that the random noise injection $g(t)$ remains identical, but the deterministic drift has been modified by a crucial new vector field: $\nabla_\mathbf{x} \log p_t(\mathbf{x})$.
This term is the score function—the gradient of the log-density of the noisy data distribution at time $t$.
Why the Score Function is the Magic Key
Imagine a drop of ink dispersing in a glass of water. If you take a snapshot halfway through the dispersion, the ink particles are scattered, but they are still denser near the center than at the edges. The score function is a vector field where every arrow points in the direction of the local increase in ink density.
By subtracting the score function from the forward drift, the reverse SDE counteracts the chaotic dispersion of the noise. It provides a directional guide that coaxes the random noise particles back toward the high-density regions of the original data manifold.
Because the true score function is unknown, we train a neural network (a score estimator) to approximate it using denoising score matching. As shown by Chen et al., as long as the neural network learns this score function accurately, the reverse SDE is guaranteed to converge closely to the target data distribution.
The Power of Unification
Before the continuous-time SDE framework, the generative modeling landscape was fragmented. Denoising Diffusion Probabilistic Models (DDPMs) and Score-Based Generative Models (such as Score Matching with Langevin Dynamics, or SMLD) were viewed as distinct methodologies with different sampling rules and objectives.
Song et al. demonstrated that these models are merely different discrete approximations of the exact same continuous-time SDEs:
- Variance Exploding (VE) SDE: Corresponds to SMLD, where noise is added continuously without scaling down the data, causing the variance to grow toward infinity.
- Variance Preserving (VP) SDE: Corresponds to DDPM, where the data is scaled down as noise is added, keeping the total variance bounded and equal to one.
This unification extends far beyond these two models. For instance, Xue et al. proved that Bayesian Flow Networks (BFNs)—which model data through continuous updates of a clean data distribution's parameters—are also mathematically equivalent to score-based diffusion when cast as SDEs.
Furthermore, the continuous formulation reveals that the choice of noise distribution during training is surprisingly flexible. Li demonstrated that discrete random walks using non-Gaussian increments (such as Laplace or Uniform noise) ultimately converge to the exact same continuous-time SDE as their step size approaches zero. This proves that the underlying continuous framework is robust to the specific micro-choices of noise injection.
Unlocking Fast Sampling and Deterministic Flows
One of the greatest practical bottlenecks of early diffusion models was speed; generating a single image required simulating thousands of sequential discrete steps. The continuous-time framework elegantly solves this.
As surveyed by Croitoru et al. and He et al., once a generative process is defined as a continuous differential equation, you are no longer locked into the discrete steps used during training. Instead, you can leverage decades of advanced numerical analysis and apply off-the-shelf, adaptive step-size SDE solvers. These solvers dynamically adjust their step sizes based on the curvature of the trajectory, reducing the required sampling steps by orders of magnitude without sacrificing sample quality.
Additionally, every continuous-time SDE possesses an associated deterministic counterpart known as the Probability Flow ODE. This ordinary differential equation shares the exact same marginal probability distributions ($p_t(x)$) as the SDE at every point in time, but strips away the stochastic noise during generation. This allows for:
- Exact Likelihood Computation: Because the trajectory is deterministic, you can compute the exact log-likelihood of a data point using standard change-of-variables formulas.
- Perfect Inversion: You can take a real image, encode it deterministically into a unique noise vector, and decode it back perfectly, which is highly valuable for image editing and solving inverse problems like MRI reconstruction (Fan et al.).
Broadening the Domain
By abstracting diffusion away from discrete pixel manipulations and into continuous vector fields, the SDE framework allows generative modeling to be applied to highly unconventional domains:
- Infinite Dimensions: Franzese et al. showed how time reversal for SDEs unifies generative techniques in infinite-dimensional function spaces, allowing models to generate continuous functional data rather than fixed-resolution grids.
- Discrete Graphs: Graphs are inherently discrete structures where adding fractional noise is non-trivial. Petersen et al. bypassed this by taking the infinitesimal limit of sequential Bayesian updates, mapping discrete graph generation directly into a continuous-time SDE over posterior distributions.
- Physical Systems: Zhang et al. utilized SDEs to model the thermodynamic relaxation of molecules, evolving random noise into stable, low-energy 3D molecular conformations by mimicking physical diffusion.
Ultimately, treating diffusion as a continuous-time stochastic process changes the fundamental problem of generative modeling from "how do we denoise an image step-by-step?" to "how do we learn a vector field that smoothly guides noise back to data?" This shift provides both the mathematical rigor needed to prove convergence and the flexibility needed to power modern generative AI.
Key Foundational Work
Partially Observable SDE Models for Image Sequence Recognition Tasks (2000) — contributed 1 concept elements to this direction.
Markov Random Walk Representations with Continuous Distributions (2003) — contributed 1 concept elements to this direction.