Abstract
This paper establishes the first rigorous mathematical framework for the convergence of denoising diffusion models (DDMs) when data is supported on a low-dimensional manifold. By utilizing the Wasserstein distance ($W_1$), the study derives quantitative error bounds that account for truncation, score approximation, and discretization. The findings demonstrate that DDMs can effectively sample from structured, high-dimensional data distributions, with convergence rates that improve significantly under manifold convexity and exhibit linear dependence on ambient dimensionality.
Introduction
Denoising diffusion models (DDMs) have achieved remarkable empirical success in generating high-fidelity images, audio, and video. However, a significant gap remains between their practical performance and theoretical understanding. Most existing convergence proofs assume that the target data distribution possesses a smooth density over the entire ambient space. In reality, high-dimensional data typically adheres to the Manifold Hypothesis, concentrating on low-dimensional structures. Under these conditions, standard density-based assumptions fail: the "score" (the gradient of the log-density) explodes as it approaches the manifold, rendering previous theoretical analyses vacuous.
This work bridges this gap by providing a rigorous analysis of DDMs in the manifold setting, offering quantitative bounds that explain how discretization, training error, and manifold geometry influence sample quality.
Theoretical Framework
The generative process is modeled as the time-reversal of a forward Ornstein–Uhlenbeck (OU) noising process. To address the challenges posed by the manifold hypothesis, the paper introduces several critical technical components:
- Wasserstein Distance ($W_1$): Unlike Total Variation or KL divergence, which can become singular when distributions do not overlap, the Wasserstein distance (or "Earth Mover's distance") provides a stable and meaningful metric for comparing distributions supported on different manifolds.
- Truncation Strategy: To mitigate the "exploding score" problem near the manifold, the backward integration process is truncated at a small time $\epsilon > 0$. This prevents the model from attempting to resolve the infinite gradients at the manifold boundary.
- Exponential Integrator (EI) Discretization: The analysis focuses on a specific discretization scheme that handles the linear drift of the diffusion process more effectively than standard Euler-Maruyama methods, mirroring practical implementations like DDPM.
- Stochastic Flow Analysis: The proof utilizes stochastic flows and interpolation formulae to track how errors—introduced by score approximation and finite step sizes—propagate through the backward process to the final generated sample.
Methodology and Validation
As a theoretical investigation, the paper validates its assumptions using geometrically controlled synthetic examples rather than standard image benchmarks. These include:
- Dirac Masses: Used to analyze score behavior near singular points.
- Concentric Circles: Providing a visualization of trajectories and score errors on a 1D manifold embedded in 2D space.
- Hypercubes: Employed to verify conditions under which the Hessian (curvature) of the log-density remains manageable.
Key Insights
The Role of Score Explosion
The paper posits that the norm of the score blowing up as $t \to 0$ is a fundamental characteristic of the manifold hypothesis rather than a numerical failure. By incorporating a $1/\sigma_t^2$ growth rate directly into the convergence assumptions, the theory aligns with the physical reality of the data.
Geometric Influence on Convergence
A pivotal finding is the impact of manifold geometry on error bounds. If the manifold $\mathcal{M}$ is convex, the dependence on the truncation parameter $\epsilon$ improves from exponential ($e^{1/\epsilon}$) to polynomial ($1/\epsilon^\Gamma$). This suggests that the "shape" of the data distribution significantly dictates the ease of sampling.
Dimension Independence
The derived error bounds depend only linearly on the ambient dimension $d$. This indicates that diffusion models are inherently efficient at handling high-dimensional data, provided the underlying manifold is well-behaved, explaining their success in tasks like high-resolution image synthesis.
Main Results
The central contribution is Theorem 1, which provides a quantitative bound on the $W_1$ distance between the true data distribution and the model's output. The error is decomposed into four distinct components:
- Truncation Error: The cost of stopping the process at $\epsilon$.
- Initialization Error: The mismatch between the Gaussian prior and the actual distribution at the end of the forward process.
- Score Approximation Error: The error introduced by the neural network's inability to perfectly learn the score.
- Discretization Error: The error resulting from finite time-stepping during the sampling process.
The theorem proves that as total time $T \to \infty$, step size $\delta \to 0$, score error $M \to 0$, and truncation $\epsilon \to 0$, the model's distribution converges exactly to the target data distribution.
Limitations and Future Directions
While this work provides a foundational proof of convergence, several areas for future research remain. The current general (non-convex) case yields an exponential dependency on $1/\epsilon$, which may be overly pessimistic compared to empirical results. Additionally, the theory assumes the manifold is compact (bounded), leaving the analysis of heavy-tailed or infinite-support distributions for future work. Finally, while the paper addresses distributional convergence, it does not explicitly quantify sample diversity or the risk of memorization in finite-sample regimes.
Summary updated