Abstract
Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum.
Introduction: The Likelihood Challenge
In generative modeling, a long-standing trade-off has existed between perceptual quality and statistical likelihood. Autoregressive models (e.g., PixelCNN) have been the gold standard for likelihood estimation—the ability to precisely estimate the probability density of data, which is essential for tasks like lossless compression. Conversely, Diffusion models have emerged as leaders in perceptual synthesis, generating realistic images while historically lagging in formal likelihood benchmarks.
Variational Diffusion Models (VDMs) address this gap, answering whether diffusion models can excel as likelihood-based models. To achieve this, the framework overcomes three primary hurdles: the limitations of fixed noise schedules, numerical instability in discrete-time likelihood calculations, and the difficulty of capturing the fine-scale pixel details required for high likelihood scores.
Methodology: SNR-based Optimization
VDMs frame the diffusion process as a hierarchical Variational Autoencoder (VAE). The key technical innovations include:
- Learnable Noise Schedule: Instead of relying on hand-tuned linear or cosine schedules, VDMs parameterize the noise schedule using a monotonic neural network, allowing the model to learn the optimal degradation path.
- SNR-based VLB: The authors derive a simplified expression for the Variational Lower Bound (VLB) based on the Signal-to-Noise Ratio (SNR). This formulation is more intuitive and numerically stable than previous discrete-time objectives.
- Continuous-Time Analysis: In the limit as the number of diffusion steps $T \to \infty$, the VLB becomes invariant to the specific noise schedule shape, depending only on the SNR at the endpoints. This allows researchers to "warp" the schedule during training to minimize estimator variance, significantly accelerating optimization.
- Fourier Features: To capture high-frequency pixel-level details, the denoising model incorporates Fourier features (sinusoidal transformations of the input), which act as a "magnifying glass" for the last mile of pixel precision.
Empirical Evaluation and Results
The model was evaluated on standard image density estimation benchmarks, including CIFAR-10 and downsampled ImageNet ($32 \times 32$ and $64 \times 64$). VDMs achieved new state-of-the-art (SOTA) bits-per-dimension (BPD) scores, surpassing autoregressive models that had dominated these benchmarks for years.
Beyond raw scores, VDMs reached these benchmarks up to 10x faster in wall-clock time than previous SOTA transformers. Furthermore, the authors demonstrated a practical "bits-back" compression scheme, achieving lossless compression rates very close to the theoretical optimum.
Key Insights and Future Directions
A central insight of this work is that the entire diffusion process can be summarized by the SNR at each timestep. This reveals that many disparate diffusion formulations are mathematically equivalent up to a rescaling of the data. The discovery of schedule invariance in continuous time further simplifies the design space, allowing schedules to be optimized for training efficiency rather than just model capacity.
While VDMs represent a leap forward in likelihood modeling, challenges remain. Generating samples still requires sequential steps, making them slower than GANs or one-step VAEs for real-time synthesis. Additionally, while the BPD scores are SOTA, closing the small remaining gap between the theoretical VLB and actual bits-back codelength remains a target for future implementation improvements.
Summary updated