Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach

Minyoung Kim, Vladimir PavlovicarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel "Gaussian Process Variational Autoencoder" (GPVAE) that significantly reduces the accuracy gap between standard VAEs and more computationally intensive methods. By modeling the VAE's inference network as a Bayesian Gaussian Process, GPVAE achieves better posterior approximations with faster inference times than existing semi-amortized approaches.

What Problem Does This Paper Address?

Variational Autoencoders (VAEs) are powerful generative models that use an "amortized inference network" (encoder) to quickly approximate the posterior distribution of latent variables. This speed comes at a cost: the amortized posterior is often less accurate than what can be achieved with instance-wise variational optimization (like Stochastic Variational Inference or SVI). This "amortization error" can degrade the VAE's performance.

Existing solutions to this problem fall into two camps:

  1. Increasing encoder capacity: Using more complex inference models (e.g., flow-based models) can improve accuracy but often makes training difficult.
  2. Semi-amortized approaches: These methods use the VAE's encoder to get a good initial guess and then perform a few SVI steps at test time. While this improves accuracy, it introduces significant computational overhead during inference.

The authors aim to bridge this gap by achieving both improved accuracy and fast inference, without the drawbacks of previous methods.

Methods

The core innovation is treating the mean and variance functions of the VAE's variational posterior distribution not as fixed neural network outputs, but as random Gaussian Processes (GPs).

Here's how it works:

  • Bayesian Treatment of Inference: Instead of a deterministic encoder network $q(z|x; \theta)$, they model the inference network's parameters themselves as random variables. Specifically, the mean and standard deviation functions of the variational posterior are modeled as independent Gaussian Processes, $f(\cdot)$ and $h(\cdot)$. This allows them to capture the uncertainty in the posterior approximation.
  • Gaussian Process Priors: The functions $f$ and $h$ are given GP priors. The mean functions of these GPs can be modeled by deep neural networks (similar to standard VAE encoders), while the covariance functions are defined by kernel functions, which can also be learned.
  • Linear Deep Kernel Trick: To make GP inference computationally tractable, they employ a "linear deep kernel trick." This represents the GP functions as linear combinations of learned feature maps, allowing them to be treated as Bayesian neural networks.
  • Marginalized Encoder: The GP posterior over the inference network parameters is marginalized out to obtain a final, amortized posterior distribution $q(z|x)$. This process is computationally efficient, involving a single feed-forward pass.
  • Uncertainty Quantification: A key benefit is that the GP framework naturally provides a measure of uncertainty in the posterior approximation. The variance of the GP functions can indicate how "Gaussian-like" or how difficult the true posterior is to approximate.

Datasets

The proposed GPVAE model was evaluated on the following standard benchmark datasets:

  • MNIST
  • OMNIGLOT
  • CIFAR10
  • SVHN
  • CelebA

Key Insights and Ideas

  • Modeling Approximation Error as Noise: The authors frame the deviation of the VAE's amortized posterior from the true posterior as random noise. By modeling this noise with GPs, they can account for it in a principled Bayesian way.
  • Principled Uncertainty Quantification: The GP framework allows the model to explicitly quantify the uncertainty in its posterior approximation. This uncertainty is shown to correlate with the difficulty of reconstructing samples and the non-Gaussianity of the true posterior.
  • Speed-Accuracy Trade-off Reimagined: The GPVAE achieves a better balance between inference speed and posterior accuracy compared to existing methods. It avoids the test-time gradient computations of semi-amortized methods while outperforming standard VAEs.
  • GP as a Bayesian Neural Network Extension: The paper highlights how GPs can be viewed as a form of Bayesian neural network, particularly when using deep kernel learning, offering a principled way to incorporate uncertainty.

Key Results

  • Improved Test Log-Likelihood: The GPVAE consistently achieved higher test log-likelihood scores across all tested datasets and latent dimensions compared to standard VAEs, semi-amortized VAEs (SA), and flow-based models (IAF, HF). For example, on MNIST with $DIM(z)=50$, GPVAE achieved a log-likelihood of 1212.9, outperforming the best competing methods.
  • Faster Inference: Compared to semi-amortized methods (SA), the GPVAE's inference time was significantly faster. While flow-based models (IAF) had comparable inference times, GPVAE achieved better accuracy. For instance, on CelebA with $DIM(z)=50$, GPVAE's per-batch inference time was 9.2 ms, while SA (8) took 53.8 ms.
  • Uncertainty Correlation: Figure 1 demonstrates that higher GP-derived uncertainty correlates with more complex, multi-modal true posteriors and poorer reconstructions, providing a useful diagnostic tool.

Limitations and Future Work

  • Computational Complexity: While faster than semi-amortized methods, the GP inference still involves matrix operations that can be computationally intensive, especially with high-dimensional latent spaces. The authors mention that further optimization of these operations is a direction for future work.
  • Scalability: The paper notes that exact GP inference can be prohibitive for very large datasets due to cubic complexity. While they use a linear deep kernel trick to mitigate this, further research into scalable approximate inference for GPs in this context could be beneficial.
  • Future Applications: The authors suggest applying GPVAE to domains with structured data, such as sequences and graphs, which would require adapting the GP kernels and feature mappings.

Summary updated

Abstract

Variational autoencoder (VAE) is a very successful generative model whose key element is the so called amortized inference network, which can perform test time inference using a single feed forward pass. Unfortunately, this comes at the cost of degraded accuracy in posterior approximation, often underperforming the instance-wise variational optimization. Although the latest semi-amortized approaches mitigate the issue by performing a few variational optimization updates starting from the VAE's amortized inference output, they inherently suffer from computational overhead for inference at test time. In this paper, we address the problem in a completely different way by considering a random inference model, where we model the mean and variance functions of the variational posterior as random Gaussian processes (GP). The motivation is that the deviation of the VAE's amortized posterior distribution from the true posterior can be regarded as random noise, which allows us to take into account the uncertainty in posterior approximation in a principled manner. In particular, our model can quantify the difficulty in posterior approximation by a Gaussian variational density. Inference in our GP model is done by a single feed forward pass through the network, significantly faster than semi-amortized methods. We show that our approach attains higher test data likelihood than the state-of-the-arts on several benchmark datasets.
BibTeX
@misc{kim2021reducing,
  author = {Minyoung Kim and Vladimir Pavlovic},
  title = {{Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach}},
  year = {2021},
  eprint = {2102.03151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.03151},
}
Figure