Recursive Inference for Variational Autoencoders

Minyoung Kim, Vladimir Pavlovic

Introduction

This paper introduces Recursive Mixture Estimation (RME), a novel method designed to boost the accuracy of Variational Autoencoders (VAEs) without sacrificing their signature speed. By iteratively adding new components to a mixture-based encoder using a functional gradient approach, the authors achieve state-of-the-art posterior approximation that remains a fast, single-pass neural network at test time.

The Problem: Speed vs. Accuracy in VAEs

In the realm of VAEs, there is a classic tug-of-war between speed and accuracy. Standard VAEs use "amortized inference"—a single encoder network that predicts latent variables for any input. While lightning-fast, this often results in an "amortization error," where the predicted posterior is a poor fit for the true, complex distribution of the data.

To fix this, researchers often turn to "semi-amortized" methods that fine-tune the encoder's output using gradient descent at test time. While more accurate, these iterative gradient update procedures are painfully slow and highly sensitive to hyperparameters like step size. The authors tackle a specific question: Can we achieve the high accuracy of iterative optimization while maintaining the "one-and-done" speed of amortized inference?

Methods: Recursive Mixture Inference

To solve this, the authors propose building a mixture inference model. Instead of a single Gaussian distribution, the encoder produces a mixture of distributions. The "secret sauce" lies in how this mixture is constructed:

  1. Recursive Augmentation: Rather than training a mixture end-to-end—which often leads to "component collapse" where all components become identical—RME adds components one by one.
  2. Functional Gradients: The model uses a greedy optimization strategy. When adding a new component, the objective is twofold:
  3. Improve the ELBO: The new component must explain the data well.
  4. Maximize Diversity: The new component is explicitly pushed to be different from the existing mixture, maximizing the KL divergence between the new component and the current mixture.
  5. Bounded KL (BKL): To prevent training from becoming numerically unstable (as KL divergence can go to infinity), they introduce a barrier or cap on the divergence term.
  6. Amortized Mixing: Unlike previous "boosted" methods that use fixed weights, RME uses a neural network to predict the mixing proportions based on the input, allowing the model to dynamically decide which mixture components are most relevant for a specific image.
Comparison of Recursive Mixture vs. Conventional Mixture in capturing complex posteriors over iterations.

Key Insights

  • Diversity is a Training Signal: Simply adding more capacity (more neurons or components) isn't enough. By explicitly rewarding the model for finding new regions of the latent space that the current model misses, RME effectively "fills in the gaps" of the posterior.
  • Avoiding the "Blind" Mixture Trap: Standard mixture models often fail because they are trained all at once. The recursive approach acts as a curriculum, making the optimization landscape much easier to navigate.
  • Amortization is Still King: You don't need to do optimization at test time if your training-time "boosting" is smart enough. This preserves the VAE's utility for real-time applications.

Key Results

The model was evaluated on four standard generative modeling benchmarks: MNIST, OMNIGLOT, SVHN, and CelebA.

  • Superior Accuracy: RME consistently outperformed standard VAEs, semi-amortized VAEs (SA), and even high-capacity flow-based models (like IAF and Householder Flows) in terms of test log-likelihood across all datasets.
  • Massive Speedup: At test time, RME is significantly faster than semi-amortized methods. For example, on the OMNIGLOT dataset, RME takes roughly 5.6ms per batch, while a semi-amortized VAE with 8 gradient steps takes 65.7ms—a ~12x speedup for better accuracy.
  • Robustness: The Bounded KL strategy proved more stable and effective than the entropy regularization used in previous "Boosted Variational Inference" (BVI) frameworks.
Test log-likelihood comparison across benchmark datasets, highlighting RME's superior performance.

Limitations and Future Work

  • Manual Mixture Order: Currently, the user must pre-define the number of mixture components. Future work could involve "stopping criteria" to automatically determine when adding more components yields diminishing returns.
  • Computational Cost at Training: While test-time inference is fast, the recursive training process is more computationally expensive than a standard VAE because it involves multiple stages of augmentation.
  • Structured Data: The current work focuses on image data. The authors express interest in extending this recursive logic to more complex structures like sequences (video/text) and graphs (molecules).

Summary updated

Abstract

Inference networks of traditional Variational Autoencoders (VAEs) are typically amortized, resulting in relatively inaccurate posterior approximation compared to instance-wise variational optimization. Recent semi-amortized approaches were proposed to address this drawback; however, their iterative gradient update procedures can be computationally demanding. In this paper, we consider a different approach of building a mixture inference model. We propose a novel recursive mixture estimation algorithm for VAEs that iteratively augments the current mixture with new components so as to maximally reduce the divergence between the variational and the true posteriors. Using the functional gradient approach, we devise an intuitive learning criteria for selecting a new mixture component: the new component has to improve the data likelihood (lower bound) and, at the same time, be as divergent from the current mixture distribution as possible, thus increasing representational diversity. Although there have been similar approaches recently, termed boosted variational inference (BVI), our methods differ from BVI in several aspects, most notably that ours deal with recursive inference in VAEs in the form of amortized inference, while BVI is developed within the standard VI framework, leading to a non-amortized single optimization instance, inappropriate for VAEs. A crucial benefit of our approach is that the inference at test time needs a single feed-forward pass through the mixture inference network, making it significantly faster than the semi-amortized approaches. We show that our approach yields higher test data likelihood than the state-of-the-arts on several benchmark datasets.
BibTeX
@inproceedings{kim2020recursive,
  author = {Minyoung Kim and Vladimir Pavlovic},
  title = {{Recursive Inference for Variational Autoencoders}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=0NejrOaFoZS},
}
Figure