Coupled Variational Bayes via Optimization Embedding

Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, Le Song

This NeurIPS 2018 paper introduces Coupled Variational Bayes (CVB), a framework that reimagines how we approximate complex probability distributions. By turning the process of optimization into a structural component of the model itself, the authors bypass some of the most persistent computational bottlenecks in Bayesian deep learning.

The Problem: The Variational Inference Tug-of-War

In Bayesian modeling, we often deal with "latent variables"—hidden factors that explain our observed data. To use these models effectively, we need the posterior distribution (the probability of hidden factors given the observed data). Because the true posterior is usually impossible to calculate exactly, we use Variational Inference (VI) to find a simpler "proxy" distribution that is close to the real thing.

Researchers typically face a three-way tug-of-war when designing these proxies:

  1. Flexibility: Simple proxies (like standard Gaussians) are easy to compute but fail to fit complex, multi-modal data well.
  2. Computation: Advanced proxies (like Normalizing Flows) are highly flexible but require calculating "Jacobian determinants." This is mathematically expensive and severely restricts the types of neural network architectures you can use.
  3. Sample Efficiency: Most methods use a separate "inference network" to predict the proxy distribution. This adds a massive number of new parameters that must be learned from scratch, which often slows down training.

The CVB Solution: Optimization as Inference

The authors solve these challenges by introducing two key innovations:

1. The Primal-Dual View of the ELBO

The Evidence Lower Bound (ELBO) is the standard metric for how well our proxy fits the true distribution. Usually, calculating this for complex transformations requires the change-of-variables formula and its dreaded Jacobian determinant. The authors use a primal-dual reformulation (based on Fenchel duality) that swaps the Jacobian requirement for a "dual function" ($\nu$). This mathematical pivot allows them to use any transformation, even non-invertible ones, without a computational penalty.

2. Optimization Embedding

Instead of building a separate neural network to act as the proxy distribution, CVB uses the path of an optimization algorithm itself.

  • The process starts with a simple noise distribution.
  • It applies $T$ steps of Mirror Descent (a generalized version of gradient descent) to "push" that noise toward the high-probability regions of the model.
  • The final state after $T$ steps is the variational distribution.

Because these optimization steps use the gradients of the model itself, the inference process is naturally "coupled" to the model parameters. When the model updates, the inference process automatically adapts.

Key Results and Insights

The authors evaluated CVB across multiple benchmarks, including synthetic datasets, MNIST (handwritten digits), and CelebA (celebrity faces), yielding several impressive results.

Superior Flexibility on Synthetic Data: In synthetic tests designed to evaluate multi-modal distribution matching, a standard VAE produced a simple, overlapping "blob" of probability. In contrast, CVB was able to sharply divide the latent space into complex shapes that perfectly matched the data's underlying structure.

Comparison of latent space clustering between vanilla VAE and CVB.

Faster Convergence on MNIST: By coupling the inference to the model parameters, CVB learns much more efficiently. On the MNIST dataset, CVB reached a better (lower) test objective significantly faster than standard VAEs, Inverse Autoregressive Flows (IAF), or Adversarial Variational Bayes (AVB).

Test objective convergence on MNIST (r=32).

High-Quality Generation: The model successfully learned to reconstruct and generate realistic faces on the CelebA dataset, proving that the optimization embedding approach scales effectively to high-dimensional, real-world data.

Theoretical Insights

  • Optimization as a Function Class: The most profound idea in this paper is that an algorithm's update rule can be viewed as a sophisticated, non-parametric function. By "unrolling" an optimizer, you get a highly flexible transformation for free.
  • The Gradient Flow Connection: The paper proves that in the limit (as optimization steps become infinitesimal), this embedding follows a "gradient flow" in the space of measures. This provides a theoretical guarantee that the method can, in principle, recover the exact true posterior.

Limitations and Future Work

While powerful, CVB is not without its challenges:

  • Dual Function Complexity: The method relies on a dual function $\nu$ parameterized by a neural network. If this network is too small or poorly trained, it can introduce bias into the gradient estimates.
  • Hyperparameter Sensitivity: Like any "unrolled" algorithm, choosing the right number of steps ($T$) and step size ($\eta$) is crucial. Too few steps limit flexibility, while too many slow down computation.

The authors suggest this "optimization embedding" is a general-purpose tool. Future work could involve applying it to Generative Adversarial Networks (GANs) or reinforcement learning to see if the same coupling benefits apply across broader domains.

Summary updated

Abstract

Variational inference plays a vital role in learning graphical models, especially on large-scale datasets. Much of its success depends on a proper choice of auxiliary distribution class for posterior approximation. However, how to pursue an auxiliary distribution class that achieves both good approximation ability and computation efficiency remains a core challenge. In this paper, we proposed coupled variational Bayes which exploits the primal-dual view of the ELBO with the variational distribution class generated by an optimization procedure, which is termed optimization embedding. This flexible function class couples the variational distribution with the original parameters in the graphical models, allowing end-to-end learning of the graphical models by back-propagation through the variational distribution. Theoretically, we establish an interesting connection to gradient flow and demonstrate the extreme flexibility of this implicit distribution family in the limit sense. Empirically, we demonstrate the effectiveness of the proposed method on multiple graphical models with either continuous or discrete latent variables comparing to state-of-the-art methods.
BibTeX
@inproceedings{dai2018coupled,
  author = {Bo Dai and Hanjun Dai and Niao He and Weiyang Liu and Zhen Liu and Jianshu Chen and Lin Xiao and Le Song},
  title = {{Coupled Variational Bayes via Optimization Embedding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
}
Figure