Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang

Introduction

Diffusion Probabilistic Models (DPMs) have established themselves as the gold standard for high-quality image generation. However, their impressive performance comes with a significant "speed limit": generating a single image typically requires iterating through thousands of noise-removal timesteps.

Analytic-DPM introduces a training-free mathematical shortcut to calculate the optimal reverse variance in diffusion models using their existing score functions. By replacing handcrafted or learned variances with these analytic estimates, the framework achieves up to an 80x speedup in sampling while simultaneously improving the model's log-likelihood.

The Speed Limit of Diffusion Models

A central challenge in the reverse process of diffusion is determining the variance at each timestep. Historically, most models have relied on "handcrafted" values (fixed constants) or required training a separate, computationally expensive neural network just to predict this variance.

Suboptimal variance leads to two primary issues:

  1. Poor density estimation: Reflected in lower log-likelihood scores.
  2. Inefficiency: The need for very long trajectories (thousands of steps) to produce clear, high-quality samples.

The authors of Analytic-DPM sought a way to find the "perfect" variance without the overhead of extra training.

The Analytic-DPM Framework

The core breakthrough of the paper is a mathematical proof demonstrating that the optimal reverse variance actually has an analytic form. It can be calculated directly if you know the "score function"—the gradient of the log-density, which the diffusion model already learns during its standard training phase.

Building on this, the authors introduce the Analytic-DPM framework, which operates on three main pillars:

  1. Analytic Estimation: It uses a pretrained score-based model and the Monte Carlo method to estimate the optimal variance and KL divergence at each step.
  2. Variance Bounding: Because pretrained score models are not perfect and can introduce bias, the authors derived theoretical lower and upper bounds for the variance. They use these bounds to "clip" the estimates, preventing errors in the score model from destabilizing the variance.
  3. Trajectory Optimization: A dynamic programming algorithm is employed to find the most efficient "path" of timesteps (e.g., jumping from 1000 steps down to 25 or 50) that minimizes the total error.
Comparison of variance estimates and bounds over timesteps, demonstrating how the analytic variance is positioned relative to traditional upper and lower bounds.

Key Insights

  • The Moment Matching Connection: The authors realized that minimizing the KL divergence in a diffusion model is equivalent to "moment matching." To get the best Gaussian approximation of the reverse step, one simply needs to match the mean and variance of the true distribution.
  • The Score-Variance Link: The critical discovery is that the variance of the reverse process is intrinsically linked to the expected squared norm of the score function. This means the information needed to optimize the variance is already "hidden" inside the pretrained model.
  • Training-Free Efficiency: Unlike previous methods that required learning a variance network (such as Improved DDPM), Analytic-DPM is entirely "plug-and-play." You can take a model someone else has already trained and immediately make its inference faster and more accurate.

Empirical Results and Speedups

The researchers evaluated Analytic-DPM on standard image generation benchmarks, including CIFAR10, CelebA 64x64, and ImageNet 64x64. The results demonstrated state-of-the-art efficiency:

  • Massive Speedups: The method achieves a 20x to 80x speedup compared to original full-trajectory models. For example, it can reach a comparable FID (Frechet Inception Distance, an image quality metric) in just 50 steps that previously took 1000.
  • Better Likelihoods: Analytic-DPM consistently improved the negative log-likelihood (NLL) across all tested models and datasets, proving that the previously used handcrafted variances were indeed suboptimal.
  • SOTA Efficiency: In comparisons with other fast-sampling methods like DDIM, Analytic-DPM often required the fewest number of timesteps to reach high-quality results.
FID scores across different datasets and timesteps. Analytic-DPM and Analytic-DDIM consistently achieve lower (better) FID scores at highly compressed trajectories (e.g., 25-50 steps) compared to baselines.

Limitations and Future Directions

While powerful, the framework does have limitations. The accuracy of the analytic estimate is inherently tied to the quality of the pretrained score model. If the model's score estimates are highly biased, the variance estimates will suffer, although the authors' bounding and clipping strategy helps mitigate this.

Additionally, the authors noted an interesting "metric tension": the optimal trajectory for log-likelihood isn't always the best for FID (visual quality). This suggests that mathematical optimality doesn't always perfectly align with human visual preference.

Future work may explore applying this framework to other data modalities, such as speech or text, and investigating how to incorporate these analytic variances directly into the initial training phase of future diffusion models.

Summary updated

Abstract

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose \textit{Analytic-DPM}, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a $20\times$ to $80\times$ speed up.
BibTeX
@inproceedings{bao2022analytic,
  author = {Fan Bao and Chongxuan Li and Jun Zhu and Bo Zhang},
  title = {{Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=QPlt66PftU},
}
Figure