Abstract
Variational Autoencoders (VAEs) are powerful generative models but are often limited by the "Gaussian bottleneck"—the assumption that latent variables follow a simple multivariate Gaussian distribution. This mismatch between the prior and the true data complexity leads to blurry samples and low fidelity. This paper proposes the Diffusion Prior, which replaces the standard Gaussian with a Denoising Diffusion Probabilistic Model (DDPM) in the latent space. By jointly training the VAE and the diffusion prior, the model captures complex latent structures more effectively. Experimental results on CelebA and CIFAR10 demonstrate that Diffusion Priors significantly improve image quality and diversity compared to standard VAEs and perform competitively with Normalizing Flow-based priors.
Introduction
Variational Autoencoders (VAEs) provide a structured latent space and efficient sampling, making them a staple in generative modeling. However, the standard VAE's reliance on a simple Gaussian prior often fails to model the intricate distributions of complex datasets like human faces or natural scenes. While Normalizing Flows (NFs) offer more flexibility, they are constrained by invertibility requirements and specific architectural designs. This paper explores whether the expressivity of Denoising Diffusion Probabilistic Models (DDPMs) can be leveraged within the VAE framework to overcome these limitations without the computational cost of pixel-space diffusion.
Methodology: The Diffusion Prior
The core innovation is a hybrid architecture where the prior distribution is itself a generative model—a DDPM. Unlike standard diffusion models that operate on high-resolution pixels, this model operates within the lower-dimensional latent space of the VAE.
Generative Process
Instead of sampling a latent vector $z$ directly from a Gaussian, the model initiates a reverse diffusion process. Starting from pure noise, the model iteratively "denoises" the signal over $T$ steps to produce a final latent vector $z_0$. This vector is then passed to a standard VAE decoder to generate the final image. This approach allows the model to break the Gaussian constraint and represent much more complex latent manifolds.
Joint Training and Loss
Training involves optimizing a modified Evidence Lower Bound (ELBO) that allows for the simultaneous training of the encoder, decoder, and diffusion prior. The objective function integrates the standard VAE reconstruction loss with a simplified DDPM loss term. This ensures that the encoder learns a latent representation that the diffusion prior can accurately model, while the decoder learns to reconstruct images from these refined latent vectors.
Experimental Results
The models were evaluated on the CIFAR10 and CelebA datasets across various latent dimensions (40, 100, and 200).
Qualitative Comparison
Visual results highlight the impact of the Diffusion Prior. On CelebA, images generated with the Diffusion Prior are noticeably sharper and more realistic than those from the Gaussian prior, which often produces blurry or artifact-heavy outputs. The Diffusion Prior successfully captures the high-frequency details of celebrity faces that the Gaussian bottleneck typically suppresses.
On CIFAR10, while the quantitative gains were more modest, the Diffusion Prior still produced more coherent structures compared to the baseline Gaussian prior.
Quantitative Performance
The primary metric, Fréchet Inception Distance (FID), confirmed the qualitative observations. On CelebA (latent size 100), the Diffusion Prior achieved an FID of 67.95, a dramatic improvement over the Gaussian prior's 149.4. The performance was also competitive with state-of-the-art Normalizing Flow priors, which were previously the gold standard for expressive VAE priors.
Discussion and Future Work
By operating in the latent space, the Diffusion Prior achieves high expressivity without the massive computational overhead typically associated with diffusion models. The flexibility of the DDPM architecture allows it to model complex distributions that rigid Normalizing Flows might struggle with.
While the current implementation uses simple backbone architectures based on DCGAN, future work could integrate more sophisticated decoders (like PixelCNN) or extend the approach to Hierarchical VAEs (e.g., NVAE). Such extensions could potentially combine the structural benefits of hierarchical models with the distribution-matching power of diffusion to reach new state-of-the-art results in efficient generative modeling.
Summary updated