Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models

Fan Bao, Chongxuan Li, Jun Zhu, Bo Zhang

Abstract

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20x to 80x speed up.

Introduction

Diffusion Probabilistic Models (DPMs) have emerged as a premier class of generative models, frequently surpassing GANs in image synthesis quality. However, their primary drawback is inference latency; generating a single sample typically requires sequential iterations through thousands of timesteps. A critical factor in this process is the "reverse variance"—the noise magnitude removed at each step.

Existing approaches often rely on handcrafted variance schedules or require training auxiliary neural networks to predict variance, which adds computational complexity. Analytic-DPM addresses this by demonstrating that the optimal reverse variance can be derived analytically from the score function, enabling a training-free path to high-speed inference.

Methodology

The core contribution of this work is the mathematical proof that the optimal reverse variance and the optimal KL divergence (the training objective for DPMs) possess closed-form analytic expressions dependent solely on the score function (the gradient of the log density).

Comparison of analytic variance and KL divergence terms across timesteps.

Building on this insight, the authors introduce Analytic-DPM, a plug-and-play framework characterized by three key components:

  1. Training-Free Estimation: It utilizes pretrained score-based models to estimate analytic variance via a Monte Carlo method, calculating the expected squared norm of the score.
  2. Bias Correction via Bounds: To mitigate errors from imperfect pretrained models, the authors derived theoretical data-independent upper and lower bounds for the optimal variance, clipping the estimates within these stable regions.
  3. Trajectory Optimization: A dynamic programming algorithm identifies the most efficient "short-cut" path through timesteps, minimizing the iterations required for high-quality sampling.

Empirical Evaluation

Analytic-DPM was evaluated across standard benchmarks including CIFAR-10, CelebA 64x64, ImageNet 64x64, and LSUN Bedroom. The results demonstrate that the analytic form allows models to achieve superior log-likelihoods with significantly fewer steps than traditional baselines.

Negative log-likelihood (NLL) results for Analytic-DPM compared to baselines across various timesteps (K).

Key results include:

  • Efficiency: The method provides a 20x to 80x speedup. For example, Analytic-DPM can reach a comparable FID in 50 steps that standard models require 1,000 steps to achieve.
  • Improved Likelihood: The framework consistently improves negative log-likelihood (NLL) across all datasets. On ImageNet 64x64, it achieved an NLL of 3.61, rivaling models with dedicated variance training.
  • Versatility: As a training-free method, it was successfully integrated into various pretrained architectures, including DDPM and DDIM, without architectural modifications.

Conclusion

Analytic-DPM reveals that the score function—already learned to determine the mean of the reverse step—contains all necessary information for the optimal variance. While the accuracy of the estimate depends on the quality of the pretrained score model, the derived bounds provide a robust safeguard. Future work may extend these analytic optimal variances to other modalities like speech or integrate them directly into the initial training phase of diffusion models.

Summary updated

Abstract

Diffusion probabilistic models (DPMs) represent a class of powerful generative models. Despite their success, the inference of DPMs is expensive since it generally needs to iterate over thousands of timesteps. A key problem in the inference is to estimate the variance in each timestep of the reverse process. In this work, we present a surprising result that both the optimal reverse variance and the corresponding optimal KL divergence of a DPM have analytic forms w.r.t. its score function. Building upon it, we propose Analytic-DPM, a training-free inference framework that estimates the analytic forms of the variance and KL divergence using the Monte Carlo method and a pretrained score-based model. Further, to correct the potential bias caused by the score-based model, we derive both lower and upper bounds of the optimal variance and clip the estimate for a better result. Empirically, our analytic-DPM improves the log-likelihood of various DPMs, produces high-quality samples, and meanwhile enjoys a 20x to 80x speed up.
BibTeX
@inproceedings{bao2022analytic,
  author = {Fan Bao and Chongxuan Li and Jun Zhu and Bo Zhang},
  title = {{Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=QPlt66PftU},
}
Figure