Variational Inference via $\chi$ Upper Bound Minimization

Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John Paisley, David M. Blei

Abstract

Variational inference (VI) is widely used as an efficient alternative to Markov chain Monte Carlo. It posits a family of approximating distributions $q$ and finds the closest member to the exact posterior $p$. Closeness is usually measured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this approach also has problems. Notably, it typically leads to underestimation of the posterior variance. In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we term the $\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved posterior uncertainty, and it can also be used with the classical VI lower bound (ELBO) to provide a sandwich estimate of the model evidence. We study CHIVI on three models: probit regression, Gaussian process classification, and a Cox process model of basketball plays. When compared to expectation propagation and classical VI, CHIVI produces better error rates and more accurate estimates of posterior variance.

Introduction

Variational inference (VI) has become the standard for scaling Bayesian analysis to large-scale datasets. By transforming the problem of posterior estimation into an optimization problem, VI avoids the computational bottlenecks often associated with Markov chain Monte Carlo (MCMC). However, traditional VI techniques, specifically those minimizing the Kullback-Leibler (KL) divergence from the approximation to the posterior ($D_{KL}(q || p)$), suffer from a well-documented "personality flaw": they are notoriously overconfident. This "zero-forcing" behavior leads to approximations that underestimate the true posterior variance, potentially resulting in risky decision-making in downstream applications.

In this work, we introduce CHIVI, a black-box variational inference algorithm that flips the traditional objective. By minimizing the $\chi$-divergence from the posterior to the approximation ($D_{\chi}(p || q)$), CHIVI provides superior uncertainty estimates and introduces a novel way to bound the true model evidence.

The Problem with Standard VI

In Bayesian analysis, the goal is to compute the posterior distribution $p(z|x)$. Standard VI approximates this with a simpler distribution $q$ by maximizing the Evidence Lower Bound (ELBO). Mathematically, this is equivalent to minimizing $D_{KL}(q || p)$.

Because the KL divergence is asymmetric, this specific direction penalizes $q$ heavily if it places mass where $p$ is zero. Consequently, the optimizer favors a "compact" $q$ that fits inside the modes of $p$, ignoring the tails and underestimating the overall spread. While alternatives like Expectation Propagation (EP) are "zero-avoiding" (trying to cover the entire support of $p$), they often lack convergence guarantees and do not optimize a clear global objective.

The CHIVI Approach: CUBO and Sandwiching

CHIVI (pronounced "kai-vi") addresses these issues by minimizing the $\chi$-divergence from $p$ to $q$. This choice has two profound implications:

  1. Zero-Avoiding Behavior: The $\chi$-divergence becomes infinite if $q$ is zero where $p$ is non-zero. This forces the approximation $q$ to "cover" the entire support of the posterior, leading to more realistic, often larger, variance estimates.
  2. The $\chi$ Upper Bound (CUBO): Minimizing this divergence is equivalent to minimizing an upper bound on the model evidence, which we term the CUBO.

By combining the ELBO (a lower bound) with the CUBO (an upper bound), we can create a "sandwich estimate" of the marginal likelihood. This provides a rigorous way to assess the quality of the variational approximation; the smaller the gap between the bounds, the more accurate the model selection.

The CUBO and ELBO provide a sandwich estimate that converges toward the true log marginal likelihood logp(x).

Black-Box Optimization

CHIVI is designed as a "black-box" algorithm, meaning it requires no model-specific derivations. It utilizes reparameterization gradients to optimize the parameters of $q$. To ensure these gradients remain unbiased, we optimize the exponentiated CUBO. While this introduces some variance into the optimization process, it allows CHIVI to be applied to any differentiable probabilistic model.

Experimental Results

We evaluated CHIVI across several models, including Bayesian probit regression and Gaussian process classification using datasets from the UCI Machine Learning Repository (e.g., Ionosphere, Pima, Heart).

Optimization dynamics on the Ionosphere dataset showing the convergence of the upper bound (CUBO) and lower bound (ELBO) over epochs.

As shown in the training dynamics, CHIVI successfully minimizes the upper bound while standard VI maximizes the lower bound. On the Ionosphere dataset, the gap between the two bounds narrows significantly, providing a tight interval for the true evidence.

In more complex scenarios, such as a Cox process model of NBA shot data (2015-2016), CHIVI demonstrated its strength in handling uncertainty. For players like Stephen Curry and DeMarcus Cousins, CHIVI captured significantly higher posterior uncertainty in areas of the court with sparse data. In contrast, standard KLVI was overconfident in these regions. CHIVI's estimates closely matched the "ground truth" provided by computationally intensive Hamiltonian Monte Carlo (HMC) simulations.

Conclusion

CHIVI offers a principled alternative to standard variational inference for researchers who prioritize accurate uncertainty quantification. By minimizing the $\chi$-divergence, it avoids the under-dispersion of KLVI and provides a tractable upper bound on model evidence. While the variance of the gradients remains a challenge for future research, the ability to "sandwich" the marginal likelihood represents a significant step forward in making variational methods more robust and interpretable.

Summary updated

Abstract

Variational inference (VI) is widely used as an efficient alternative to\nMarkov chain Monte Carlo. It posits a family of approximating distributions $q$\nand finds the closest member to the exact posterior $p$. Closeness is usually\nmeasured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this\napproach also has problems. Notably, it typically leads to underestimation of\nthe posterior variance. In this paper we propose CHIVI, a black-box variational\ninference algorithm that minimizes $D_{\\chi}(p || q)$, the $\\chi$-divergence\nfrom $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we\nterm the $\\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved\nposterior uncertainty, and it can also be used with the classical VI lower\nbound (ELBO) to provide a sandwich estimate of the model evidence. We study\nCHIVI on three models: probit regression, Gaussian process classification, and\na Cox process model of basketball plays. When compared to expectation\npropagation and classical VI, CHIVI produces better error rates and more\naccurate estimates of posterior variance.\n
BibTeX
@inproceedings{dieng2017variational,
  author = {Adji Bousso Dieng and Dustin Tran and Rajesh Ranganath and John Paisley and David M. Blei},
  title = {{Variational Inference via $\textbackslash\{\}chi$ Upper Bound Minimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=r1W90IWObS},
}
Figure