Diffusion models have taken the world by storm with their ability to generate stunning, high-fidelity images. However, in the world of probabilistic modeling, "looking good" isn't everything. For years, autoregressive models like PixelCNN and Transformers have reigned supreme in density estimation—the task of accurately modeling the underlying probability distribution of data.
In this paper, researchers from Google Research introduce Variational Diffusion Models (VDMs), a framework that finally allows diffusion models to challenge and surpass the dominance of autoregressive models on standard image benchmarks.
TL;DR
Variational Diffusion Models (VDMs) achieve new state-of-the-art likelihoods on image density estimation by simplifying the variational lower bound (VLB) into a function of the signal-to-noise ratio. By jointly optimizing a learnable noise schedule and incorporating Fourier features, VDMs outperform long-dominant autoregressive models while offering faster optimization and a path toward optimal lossless compression.
What Problem Does This Paper Address?
While diffusion models excel at synthesis (perceptual quality), they have traditionally struggled to match the log-likelihood scores of autoregressive models. High log-likelihood is essential for tasks like lossless data compression and rigorous density estimation.
The challenge lies in the complexity of the diffusion process. Previous methods often used fixed, hand-tuned noise schedules and struggled to capture the fine-grained pixel details necessary for high-likelihood modeling. This paper aims to bridge that gap, making diffusion models first-class citizens in the world of likelihood-based modeling.
Methods
The authors introduce several key innovations to the standard diffusion pipeline:
- Simplified Variational Lower Bound (VLB): They derive a remarkably simple expression for the VLB in terms of the Signal-to-Noise Ratio (SNR) of the diffused data. This provides a clearer theoretical understanding of what the model is actually optimizing.
- Learnable Noise Schedule: Instead of fixing the noise schedule (how quickly data turns into noise), they parameterize it using a monotonic neural network. This allows the model to learn the optimal "path" of degradation jointly with the denoising model.
- Fourier Features: To capture the fine-scale details required for high likelihoods, the authors incorporate high-frequency Fourier features into the input of the denoising neural network.
- Continuous-Time Analysis: They analyze the model in the limit of infinite diffusion steps ($T \to \infty$). They prove that in this continuous-time setting, the VLB is invariant to the specific noise schedule (except for its endpoints), which allows them to optimize the schedule specifically to minimize training variance.
Key Insights and Ideas
- SNR is the Natural Parameter: The discovery that the diffusion loss is essentially an integral of the Mean Squared Error (MSE) over the SNR range is a powerful simplification. It shows that the model's performance is fundamentally tied to how well it can denoise at different signal levels.
- Invariance and Variance Minimization: The proof that the continuous-time VLB is invariant to the noise schedule path is a "lightbulb moment." It means we can change the schedule to make training more stable and faster without changing what the model is theoretically capable of learning.
- More Steps are Always Better: The authors prove that increasing the number of diffusion steps $T$ always improves (lowers) the VLB, justifying the move toward continuous-time formulations.
Key Results
The models were evaluated on standard image density estimation benchmarks, including CIFAR-10 (with and without data augmentation) and ImageNet (downsampled to $32 \times 32$ and $64 \times 64$).
- State-of-the-Art Likelihoods: VDMs achieved a bits-per-dimension (BPD) of 2.65 on CIFAR-10 (no augmentation) and 3.40 on ImageNet 64x64, outperforming contemporary autoregressive models like the Routing Transformer.
- Faster Optimization: Despite their depth, VDMs often optimized significantly faster than their autoregressive counterparts.
- Lossless Compression: The authors demonstrated a "bits-back" compression scheme using VDMs that achieves rates very close to the theoretical optimum, proving their utility for practical data storage.
Limitations and Future Work
- Compression Complexity: While VDMs achieve near-optimal compression rates, the process is computationally expensive, requiring many forward passes of the neural network. Future work is needed to make "bits-back" coding more efficient for such deep models.
- Synthesis vs. Likelihood: The authors note that models optimized for likelihood don't always produce the most perceptually pleasing images. Tuning the model specifically for synthesis (e.g., using weighted losses) remains a separate, though related, challenge.
- Discrete vs. Continuous: While the continuous-time limit is theoretically elegant, practical implementation still requires discretization, and closing the gap between the two remains an area for further refinement.
Summary updated