Variational Inference via \chi Upper Bound Minimization

Adji Bousso Dieng, Dustin Tran, Rajesh Ranganath, John W. Paisley, David M. Blei

Introduction

Standard Variational Inference (VI) is a cornerstone of Bayesian machine learning, offering a computationally efficient way to approximate complex posterior distributions. However, it often underestimates uncertainty, leading to overconfident but inaccurate models. To address this, researchers introduce CHIVI ($\chi$-divergence Variational Inference), a black-box algorithm that minimizes the $\chi$-divergence to create an upper bound on model evidence. By combining this with the traditional lower bound, CHIVI allows researchers to "sandwich" the true model evidence and produce much more realistic estimates of posterior variance.

The Problem with Standard VI

In Bayesian modeling, finding the exact posterior distribution $p(z|x)$—how hidden variables $z$ behave given observed data $x$—is usually intractable. VI solves this by finding a simpler distribution $q$ that is "close" to $p$.

The industry standard, KLVI, minimizes the Kullback-Leibler (KL) divergence from $q$ to $p$. Unfortunately, KLVI exhibits "zero-forcing" behavior: it prefers to ignore parts of the true distribution rather than risk putting mass where the true distribution is zero. This results in "underdispersed" models that underestimate risk and variance. While alternatives like Expectation Propagation (EP) attempt to fix this, they often fail to converge and lack a clear global objective function.

CHIVI and the CUBO Method

Instead of the standard KL divergence, CHIVI minimizes the $\chi^2$-divergence from the posterior $p$ to the approximation $q$. This shift in divergence metric yields several powerful technical contributions:

  • The CUBO (Chi Upper Bound): The authors derive a new objective function that serves as a rigorous upper bound on the log marginal likelihood ($\log p(x)$).
  • Zero-Avoiding Behavior: Unlike KL divergence, the $\chi$-divergence is "zero-avoiding." It becomes infinite if $q(z)$ is zero while $p(z|x)$ is positive. This forces the approximation $q$ to cover the entire support of the posterior, preventing the overconfidence seen in standard VI.
  • Black-Box Optimization: CHIVI does not require complex, model-specific derivations. It uses reparameterization gradients on the exponentiated CUBO to ensure estimates remain unbiased during optimization.

The Sandwich Theorem and Key Results

By utilizing the standard ELBO (a lower bound) alongside the new CUBO (an upper bound), researchers can bracket the true model evidence. This "sandwich estimate" is highly valuable for model selection and evaluating the tightness of the approximation.

Sandwich plots demonstrating the convergence of the upper bound (CUBO) and lower bound (ELBO) around the true log marginal likelihood across different datasets.

The authors evaluated CHIVI across Bayesian Probit Regression, Gaussian Process (GP) Classification, and Cox Processes. The results highlight CHIVI's strengths:

  • Tight Evidence Bracketing: As optimization progresses, the gap between the ELBO and CUBO narrows, providing a reliable, bounded range for the true model evidence.
  • Lower Error Rates: On GP classification tasks—where standard VI often struggles due to light-tailed posteriors—CHIVI outperformed both the Laplace approximation and EP on benchmark datasets like Sonar and Ionosphere.
  • Better Uncertainty Estimation: To test spatial modeling, the authors applied a Cox process to NBA shot patterns from the 2015-2016 season. CHIVI produced posterior uncertainty estimates that closely matched Hamiltonian Monte Carlo (HMC, the gold standard), whereas standard VI significantly underestimated uncertainty in sparse data regions.
NBA shot charts and posterior uncertainty heatmaps comparing standard VI (KLQP), CHIVI (Chi), and HMC. CHIVI's uncertainty estimates closely mirror the HMC ground truth.

Limitations and Future Work

The primary drawback of CHIVI is computational variance. Exponentiating the CUBO bound to obtain unbiased gradients can lead to high variance during optimization. Developing more sophisticated variance reduction schemes for these types of objectives remains a critical open research problem. Additionally, while CHIVI excels at capturing variance and avoiding zero-forcing, it may be less suited for multimodal posteriors where focusing on a single mode is preferred for specific predictive tasks.

Summary updated

Abstract

Variational inference (VI) is widely used as an efficient alternative to Markov chain Monte Carlo. It posits a family of approximating distributions $q$ and finds the closest member to the exact posterior $p$. Closeness is usually measured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this approach also has problems. Notably, it typically leads to underestimation of the posterior variance. In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we term the $\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved posterior uncertainty, and it can also be used with the classical VI lower bound (ELBO) to provide a sandwich estimate of the model evidence. We study CHIVI on three models: probit regression, Gaussian process classification, and a Cox process model of basketball plays. When compared to expectation propagation and classical VI, CHIVI produces better error rates and more accurate estimates of posterior variance.
BibTeX
@inproceedings{dieng2017variational,
  author = {Adji Bousso Dieng and Dustin Tran and Rajesh Ranganath and John W. Paisley and David M. Blei},
  title = {{Variational Inference via \textbackslash\{\}chi Upper Bound Minimization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=r1W90IWObS},
}
Figure