Using Ornstein-Uhlenbeck Process to understand Denoising Diffusion Probabilistic Model and its Noise Schedules

Javier E. Santos, Yen Ting LinarXiv 2023 · DOI · arXiv

Introduction

Denoising Diffusion Probabilistic Models (DDPMs) have revolutionized generative AI, yet their performance often hinges on the "noise schedule"—the sequence of noise levels added during training. While the "cosine schedule" has become a standard empirical choice, its theoretical justification has remained elusive. This research provides a rigorous foundation by mapping DDPMs to the continuous-time Ornstein–Uhlenbeck (OU) process.

Theoretical Framework: DDPM as an OU Process

The authors demonstrate that the discrete-time Markov chain of a DDPM is mathematically equivalent to a time-homogeneous OU process sampled at non-uniform intervals. By setting the relaxation rate $\gamma=1$ and fluctuation strength $\sigma=\sqrt{2}$, the DDPM equations emerge as a discretized version of the OU process. This perspective shifts the focus from designing step-wise noise ($\beta_k$) to determining the optimal observation times ($t_k$) of a continuous process.

Deriving Noise Schedules from First Principles

Using the OU framework, the researchers derived four noise schedules based on distinct physical and information-theoretic principles:

  1. EquiVariance (EqV): Maintains constant auto-variance, resulting in a linear schedule.
  2. Coefficient of Variation (CV): Leads to a quadratic schedule.
  3. EquiEntropy (EqE): Produces an exponential-style schedule.
  4. Fisher Information (FI): Optimizing for sampling density proportional to the square root of Fisher Information leads exactly to the popular cosine schedule.
Comparison of cumulative noise ($\bar{\alpha}$) schedules derived from different heuristics. The Fisher Information (FI) heuristic corresponds to the cosine schedule.

Empirical Results

The theoretical derivations were validated on the CIFAR-10 dataset. Models were trained for 500,000 iterations using 1,000 and 4,000 diffusion steps. The results confirm that the Fisher Information (FI) schedule, which yields the cosine schedule, is not just an engineering trick but a theoretically optimal approach.

FID values on CIFAR-10 for different noise schedules at 1k and 4k steps. The FI (cosine) schedule consistently achieves the best (lowest) FID scores.

As shown in the performance metrics, the FI-based schedule consistently achieved the lowest Fréchet Inception Distance (FID), indicating superior image quality. Furthermore, this schedule demonstrated high robustness across different step counts, validating the stability of the OU-based derivation.

Conclusion and Future Work

This work demystifies the success of the cosine schedule by linking it to Fisher Information within an OU process framework. While the current model assumes independent pixel channels, it provides a "rational" starting point for future schedules that might incorporate data-dependent correlations or more complex stochastic processes.

Summary updated

Abstract

The aim of this short note is to show that Denoising Diffusion Probabilistic Model DDPM, a non-homogeneous discrete-time Markov process, can be represented by a time-homogeneous continuous-time Markov process observed at non-uniformly sampled discrete times. Surprisingly, this continuous-time Markov process is the well-known and well-studied Ornstein-Ohlenbeck (OU) process, which was developed in 1930's for studying Brownian particles in Harmonic potentials. We establish the formal equivalence between DDPM and the OU process using its analytical solution. We further demonstrate that the design problem of the noise scheduler for non-homogeneous DDPM is equivalent to designing observation times for the OU process. We present several heuristic designs for observation times based on principled quantities such as auto-variance and Fisher Information and connect them to ad hoc noise schedules for DDPM. Interestingly, we show that the Fisher-Information-motivated schedule corresponds exactly the cosine schedule, which was developed without any theoretical foundation but is the current state-of-the-art noise schedule.
BibTeX
@misc{santos2023using,
  author = {Javier E. Santos and Yen Ting Lin},
  title = {{Using Ornstein-Uhlenbeck Process to understand Denoising Diffusion Probabilistic Model and its Noise Schedules}},
  year = {2023},
  eprint = {2311.17673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.17673},
}
Figure