Diffusion Priors In Variational Autoencoders

Antoine Wehenkel, Gilles Louppe

Abstract

Variational Autoencoders (VAEs) are powerful generative models but are often limited by the "Gaussian bottleneck"—the assumption that latent variables follow a simple multivariate Gaussian distribution. This mismatch between the prior and the true data complexity leads to blurry samples and low fidelity. This paper proposes the Diffusion Prior, which replaces the standard Gaussian with a Denoising Diffusion Probabilistic Model (DDPM) in the latent space. By jointly training the VAE and the diffusion prior, the model captures complex latent structures more effectively. Experimental results on CelebA and CIFAR10 demonstrate that Diffusion Priors significantly improve image quality and diversity compared to standard VAEs and perform competitively with Normalizing Flow-based priors.

Introduction

Variational Autoencoders (VAEs) provide a structured latent space and efficient sampling, making them a staple in generative modeling. However, the standard VAE's reliance on a simple Gaussian prior often fails to model the intricate distributions of complex datasets like human faces or natural scenes. While Normalizing Flows (NFs) offer more flexibility, they are constrained by invertibility requirements and specific architectural designs. This paper explores whether the expressivity of Denoising Diffusion Probabilistic Models (DDPMs) can be leveraged within the VAE framework to overcome these limitations without the computational cost of pixel-space diffusion.

Methodology: The Diffusion Prior

The core innovation is a hybrid architecture where the prior distribution is itself a generative model—a DDPM. Unlike standard diffusion models that operate on high-resolution pixels, this model operates within the lower-dimensional latent space of the VAE.

Generative Process

Instead of sampling a latent vector $z$ directly from a Gaussian, the model initiates a reverse diffusion process. Starting from pure noise, the model iteratively "denoises" the signal over $T$ steps to produce a final latent vector $z_0$. This vector is then passed to a standard VAE decoder to generate the final image. This approach allows the model to break the Gaussian constraint and represent much more complex latent manifolds.

Joint Training and Loss

Training involves optimizing a modified Evidence Lower Bound (ELBO) that allows for the simultaneous training of the encoder, decoder, and diffusion prior. The objective function integrates the standard VAE reconstruction loss with a simplified DDPM loss term. This ensures that the encoder learns a latent representation that the diffusion prior can accurately model, while the decoder learns to reconstruct images from these refined latent vectors.

Experimental Results

The models were evaluated on the CIFAR10 and CelebA datasets across various latent dimensions (40, 100, and 200).

Qualitative Comparison

Visual results highlight the impact of the Diffusion Prior. On CelebA, images generated with the Diffusion Prior are noticeably sharper and more realistic than those from the Gaussian prior, which often produces blurry or artifact-heavy outputs. The Diffusion Prior successfully captures the high-frequency details of celebrity faces that the Gaussian bottleneck typically suppresses.

Comparison of generated samples on CelebA using Gaussian, NF, and Diffused priors.

On CIFAR10, while the quantitative gains were more modest, the Diffusion Prior still produced more coherent structures compared to the baseline Gaussian prior.

Comparison of generated samples on CIFAR10 using Gaussian, NF, and Diffused priors.

Quantitative Performance

The primary metric, Fréchet Inception Distance (FID), confirmed the qualitative observations. On CelebA (latent size 100), the Diffusion Prior achieved an FID of 67.95, a dramatic improvement over the Gaussian prior's 149.4. The performance was also competitive with state-of-the-art Normalizing Flow priors, which were previously the gold standard for expressive VAE priors.

Discussion and Future Work

By operating in the latent space, the Diffusion Prior achieves high expressivity without the massive computational overhead typically associated with diffusion models. The flexibility of the DDPM architecture allows it to model complex distributions that rigid Normalizing Flows might struggle with.

While the current implementation uses simple backbone architectures based on DCGAN, future work could integrate more sophisticated decoders (like PixelCNN) or extend the approach to Hierarchical VAEs (e.g., NVAE). Such extensions could potentially combine the structural benefits of hierarchical models with the distribution-matching power of diffusion to reach new state-of-the-art results in efficient generative modeling.

Summary updated

Abstract

Among likelihood-based approaches for deep generative modelling, variational autoencoders (VAEs) offer scalable amortized posterior inference and fast sampling. However, VAEs are also more and more outperformed by competing models such as normalizing flows (NFs), deep-energy models, or the new denoising diffusion probabilistic models (DDPMs). In this preliminary work, we improve VAEs by demonstrating how DDPMs can be used for modelling the prior distribution of the latent variables. The diffusion prior model improves upon Gaussian priors of classical VAEs and is competitive with NF-based priors. Finally, we hypothesize that hierarchical VAEs could similarly benefit from the enhanced capacity of diffusion priors.
BibTeX
@inproceedings{wehenkel2021diffusion,
  author = {Antoine Wehenkel and Gilles Louppe},
  title = {{Diffusion Priors In Variational Autoencoders}},
  booktitle = {INNF @ ICML},
  year = {2021},
  url = {https://openreview.net/forum?id=DoGRjqtgjph},
}
Figure