Diffusion models operate on a simple premise: if we can learn the "score function"—the gradient of the log-density of a data distribution—we can reverse a noise-injecting process to generate new samples. Most theoretical foundations for these models, including the convergence proofs for deterministic samplers provided by Chen et al. et al., rely on the perfect score assumption. This assumption posits that the generative model has access to the exact gradient of the data distribution at every noise level.
In practice, this assumption is never met. The score function is approximated by a neural network trained on finite samples. This approximation error is not merely a minor nuisance; it is the primary factor determining the fidelity and stability of generated samples.
The Dominance of Approximation Error
When a diffusion model fails to produce a high-quality image, researchers often look to the numerical solver. They attempt to reduce discretization error—the mathematical artifact of taking finite steps along a continuous path. However, Yu et al. et al. demonstrate that for fast sampling, the neural network's approximation error is actually the dominant source of degradation.
This error is not uniform across the generation process. Yu et al. et al. find that denoising networks exhibit significantly higher approximation errors as the process approaches "cleaner," low-noise states. While the network is adept at identifying the general structure of a signal in heavy noise, its ability to precisely estimate the score function for nearly-clean data is limited. This inaccuracy at later stages of sampling acts as a bottleneck, preventing high-order mathematical solvers from reaching their theoretical potential at low step counts.
Error Accumulation and Manifold Drift
The generation process in a diffusion model is iterative. Each step relies on the output of the previous one, creating a chain where small inaccuracies can compound. Pierret et al. et al. argue that deterministic sampling schemes, such as those using high-order ODE solvers, are particularly sensitive to these inaccuracies.
If the learned score function deviates from the true score, the sampling trajectory can be pushed away from the "data manifold"—the low-dimensional space where natural data resides. Once the trajectory drifts into these out-of-distribution regions, the neural network (which was likely not trained on such "impossible" data) provides even less accurate score estimates. Lin and Huang et al. show that the final approximation error of a probability flow ODE sampler is linearly proportional to the error of the learned score-matching function, suggesting that sample fidelity is strictly capped by the network's training quality.
Fragility versus Robustness
The choice of sampling dynamics—whether to use a deterministic Ordinary Differential Equation (ODE) or a stochastic Stochastic Differential Equation (SDE)—significantly changes how approximation errors manifest.
- Langevin Dynamics Fragility: Cao et al. et al. reveal that Langevin dynamics is highly vulnerable to score error. In high-dimensional settings, even infinitesimal errors in the estimated score can lead to samples that diverge drastically from the true target distribution.
- SDE Robustness: Conversely, Nie et al. et al. argue that the randomness inherent in SDEs acts as a "contractive" force. This stochasticity effectively pulls edited or drifting images back toward the natural data distribution, providing a buffer against the systematic biases of a learned score function.
Propagation to Downstream Tasks
The impact of score approximation error extends beyond image quality into scientific and statistical applications:
- Bayesian Computation: Heng and Douc et al. note that the accuracy of diffusion-based Bayesian inference is contingent on score quality, yet there is currently no standard method to rigorously verify the error introduced by these neural approximations.
- Intrinsic Dimension Estimation: When using diffusion models to estimate the local intrinsic dimension (LID) of data, inaccuracies in the score function can lead to physically impossible results, such as negative dimension estimates, as noted by Leung et al. et al.
- Parameter Estimation: In "blind denoising" tasks, Heurtel-Depeiges et al. et al. show that minor approximation errors in the generative prior can propagate through the inference chain, introducing systematic biases into the estimation of physical constants or noise levels.
The Gap in Theory
Current theoretical frameworks often treat score estimation errors as idealized, $L^2$-bounded inaccuracies. However, Lee et al. et al. point out that while neural networks minimize error on average, they may leave regions of high local error that sampling processes might inadvertently enter. As Lin and Huang et al. suggest, a major open challenge for the field is establishing a formal link between the actual training dynamics of neural networks and the resulting error profiles that govern sampling performance.
Key Foundational Work
Diffusion Schrödinger Bridges for Bayesian Computation (2023) — contributed 2 concept elements to this direction.
Refining Generative Process with Discriminator Guidance in Score-based Diffusion Models (2023) — contributed 2 concept elements to this direction.
Expressiveness Remarks for Denoising Diffusion Based Sampling (2023) — contributed 2 concept elements to this direction.