Error accumulation in reverse diffusion with an imperfect score: beyond path-space analyses

George Giapitzakis, Ankur Moitra, Andrej Risteski, Dhruv RohatgiarXiv 2026 · arXiv

Diffusion models have revolutionized generative AI, but their theoretical foundations often suggest a worrying trend: because they generate data through a long sequence of steps, small errors in the learned "score function" should theoretically snowball into massive failures. This phenomenon, often called exposure bias, has been a thorn in the side of researchers trying to prove why these models work so well in practice.

In their paper, "Error accumulation in reverse diffusion with an imperfect score: beyond path-space analyses," researchers from CMU, MIT, and UC Berkeley provide a more optimistic—and realistic—theoretical framework. They demonstrate that under the right geometric conditions, diffusion models don't just accumulate errors; they can actually attenuate them.

TL;DR

The authors show that the common theoretical fear of polynomial error accumulation in diffusion models is often too pessimistic. By moving beyond "path-space" analysis, they prove that for distributions satisfying a Poincaré inequality, error accumulation is significantly mitigated, scaling logarithmically rather than polynomially with the time horizon.

What Problem Does This Paper Address?

To generate an image, a diffusion model starts with pure noise and iteratively "denoises" it using a learned score function (the gradient of the log-density). Standard theoretical analyses suggest that the error in the final output grows as $\epsilon \sqrt{T}$, where $\epsilon$ is the error of the learned score and $T$ is the number of steps (or time horizon).

The reason for this pessimistic bound is "path-space analysis": theorists were essentially measuring how much the entire trajectory of the generated sample deviated from the ideal trajectory. However, in practice, we only care about the final sample. If a model makes a mistake early in the process, the subsequent denoising steps might "wash out" that error. This paper seeks to prove when and why this "self-correction" happens.

Methods

The authors employ a few key mathematical innovations to bypass the limitations of previous work:

  1. The Adjoint Method: Instead of comparing entire paths, they use an adjoint estimate to bound the discrepancy of the terminal marginals (the final output). This allows them to focus on how errors at time $t$ affect the final result at time $0$.
  2. Poincaré Inequalities: They use this geometric tool to characterize "well-behaved" distributions. A distribution satisfying a Poincaré inequality is essentially "well-connected," meaning a sampler can move across the distribution's space relatively easily without getting stuck in isolated modes.
  3. $f$-conditioned Poincaré Inequality: This is a novel, finer-grained metric introduced by the authors. It allows them to analyze specific "test statistics" (properties of the data). Even if a distribution is multimodal (which usually breaks Poincaré bounds), certain features of the data might still be learned accurately if those features don't "witness" the multimodality.
  4. Information-Theoretic Lower Bounds: They construct "multi-scale Gaussians" to prove that their logarithmic error bounds are essentially the best any algorithm could achieve.

Datasets

Not discussed in the paper. This is a purely theoretical work focused on proving mathematical guarantees and limits. No empirical experiments on standard image or text datasets were conducted.

Key Insights and Ideas

  • The "Path-Space" Fallacy: Comparing the entire generation path is too strict. Discrepancies between entire paths can be much larger than discrepancies between their final endpoints because the dynamics of the SDE can attenuate early errors.
  • Self-Correction through Smoothing: The forward diffusion process (adding noise) quickly "smooths" even a messy distribution into one that satisfies a Poincaré inequality. The reverse process benefits from this: at high noise levels, the model is working on a "nice" distribution where errors don't compound as badly.
  • Feature-Specific Accuracy: The $f$-conditioned Poincaré inequality explains why diffusion models might be great at getting the "perceptual quality" right (local features) even if they struggle with the "relative weights" of different classes (global multimodality). If a feature $f$ is "balanced" across modes, the model learns it with high accuracy.
  • Logarithmic Scaling: For distributions with a Poincaré constant $C_P$, the error accumulation factor is roughly $\sqrt{\log(C_P)}$ rather than $\sqrt{T}$. Since $T$ is often chosen to be proportional to $\log(C_P)$, this represents a significant theoretical tightening.

Key Results

  • Theorem 1.2 (Informal): For a specific property $f$, the error in the model's estimate of that property is bounded by a term that depends on the "f-conditioned Poincaré constant." If this constant is small, the error is low, regardless of how long the diffusion process runs.
  • Corollary 1.3: If the target distribution $p$ satisfies a global Poincaré inequality, the Total Variation (TV) distance between the true and generated distributions scales with $\epsilon \sqrt{\log(C_P)}$, effectively removing the direct dependence on the time horizon $T$.
  • Optimality: The authors show that their bound is nearly optimal. They prove that any sampler, given a score with error $\epsilon$, must incur at least $\Omega(\epsilon \sqrt{\log K})$ error in TV distance for a distribution with Poincaré constant $K$.

Limitations and Future Work

  • Computational Efficiency: While the authors identify an information-theoretically "optimal" way to identify the distribution $p$ from a learned score, the algorithm to do so is computationally inefficient (it involves projecting onto a family of vector fields). Whether a computationally efficient algorithm can reach the same limit remains an open question.
  • Beyond Poincaré: The current results rely heavily on the Poincaré inequality. The authors suggest that exploring other structural assumptions (like the manifold hypothesis) could further refine our understanding of error propagation.
  • Discretization Gap: The paper focuses on the continuous-time SDE. While they believe the results extend to discrete numerical solvers, the formal proof for specific discretization schemes (like DDPM or DDIM) is left for future work.

Summary updated

Abstract

Diffusion modeling is a popular method of sampling from a complex, high-dimensional distribution $p$ using learned approximations of its (noised) score functions. Generating a single sample requires simulating the reverse diffusion process, which requires many queries to the learned scores. Thus, a priori, the learning errors can easily compound. Indeed, in existing theoretical analyses, the sampling error (as measured in total variation distance to $p$) accumulates polynomially with the horizon of the stochastic process. This is a consequence of bounding the error of the output law by the error of the law of the entire reverse diffusion path. In this paper, we demonstrate that by circumventing the "path-space" analysis, the error accumulation of the diffusion sampler can be mitigated when $p$ satisfies a Poincaré inequality. Moreover, we achieve this result as a special case of a broader framework that applies to any distribution $p$, but relaxes the error metric from total variation distance to a weaker metric capturing all sample statistics that do not "witness" the multimodality of $p$. We then zoom out from analyzing the diffusion sampler to the information-theoretic question: given a learned score with some error bound, how accurately can we identify $p$? We show that, for any given bound on the Poincaré constant of $p$, the diffusion sampler already nearly achieves optimal total variation error.
BibTeX
@misc{giapitzakis2026error,
  author = {George Giapitzakis and Ankur Moitra and Andrej Risteski and Dhruv Rohatgi},
  title = {{Error accumulation in reverse diffusion with an imperfect score: beyond path-space analyses}},
  year = {2026},
  eprint = {2611.23617},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23617},
}
Figure