Abstract
Variational inference (VI) is widely used as an efficient alternative to Markov chain Monte Carlo. It posits a family of approximating distributions $q$ and finds the closest member to the exact posterior $p$. Closeness is usually measured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this approach also has problems. Notably, it typically leads to underestimation of the posterior variance. In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we term the $\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved posterior uncertainty, and it can also be used with the classical VI lower bound (ELBO) to provide a sandwich estimate of the model evidence. We study CHIVI on three models: probit regression, Gaussian process classification, and a Cox process model of basketball plays. When compared to expectation propagation and classical VI, CHIVI produces better error rates and more accurate estimates of posterior variance.
Introduction
Variational inference (VI) has become the standard for scaling Bayesian analysis to large-scale datasets. By transforming the problem of posterior estimation into an optimization problem, VI avoids the computational bottlenecks often associated with Markov chain Monte Carlo (MCMC). However, traditional VI techniques, specifically those minimizing the Kullback-Leibler (KL) divergence from the approximation to the posterior ($D_{KL}(q || p)$), suffer from a well-documented "personality flaw": they are notoriously overconfident. This "zero-forcing" behavior leads to approximations that underestimate the true posterior variance, potentially resulting in risky decision-making in downstream applications.
In this work, we introduce CHIVI, a black-box variational inference algorithm that flips the traditional objective. By minimizing the $\chi$-divergence from the posterior to the approximation ($D_{\chi}(p || q)$), CHIVI provides superior uncertainty estimates and introduces a novel way to bound the true model evidence.
The Problem with Standard VI
In Bayesian analysis, the goal is to compute the posterior distribution $p(z|x)$. Standard VI approximates this with a simpler distribution $q$ by maximizing the Evidence Lower Bound (ELBO). Mathematically, this is equivalent to minimizing $D_{KL}(q || p)$.
Because the KL divergence is asymmetric, this specific direction penalizes $q$ heavily if it places mass where $p$ is zero. Consequently, the optimizer favors a "compact" $q$ that fits inside the modes of $p$, ignoring the tails and underestimating the overall spread. While alternatives like Expectation Propagation (EP) are "zero-avoiding" (trying to cover the entire support of $p$), they often lack convergence guarantees and do not optimize a clear global objective.
The CHIVI Approach: CUBO and Sandwiching
CHIVI (pronounced "kai-vi") addresses these issues by minimizing the $\chi$-divergence from $p$ to $q$. This choice has two profound implications:
- Zero-Avoiding Behavior: The $\chi$-divergence becomes infinite if $q$ is zero where $p$ is non-zero. This forces the approximation $q$ to "cover" the entire support of the posterior, leading to more realistic, often larger, variance estimates.
- The $\chi$ Upper Bound (CUBO): Minimizing this divergence is equivalent to minimizing an upper bound on the model evidence, which we term the CUBO.
By combining the ELBO (a lower bound) with the CUBO (an upper bound), we can create a "sandwich estimate" of the marginal likelihood. This provides a rigorous way to assess the quality of the variational approximation; the smaller the gap between the bounds, the more accurate the model selection.
Black-Box Optimization
CHIVI is designed as a "black-box" algorithm, meaning it requires no model-specific derivations. It utilizes reparameterization gradients to optimize the parameters of $q$. To ensure these gradients remain unbiased, we optimize the exponentiated CUBO. While this introduces some variance into the optimization process, it allows CHIVI to be applied to any differentiable probabilistic model.
Experimental Results
We evaluated CHIVI across several models, including Bayesian probit regression and Gaussian process classification using datasets from the UCI Machine Learning Repository (e.g., Ionosphere, Pima, Heart).
As shown in the training dynamics, CHIVI successfully minimizes the upper bound while standard VI maximizes the lower bound. On the Ionosphere dataset, the gap between the two bounds narrows significantly, providing a tight interval for the true evidence.
In more complex scenarios, such as a Cox process model of NBA shot data (2015-2016), CHIVI demonstrated its strength in handling uncertainty. For players like Stephen Curry and DeMarcus Cousins, CHIVI captured significantly higher posterior uncertainty in areas of the court with sparse data. In contrast, standard KLVI was overconfident in these regions. CHIVI's estimates closely matched the "ground truth" provided by computationally intensive Hamiltonian Monte Carlo (HMC) simulations.
Conclusion
CHIVI offers a principled alternative to standard variational inference for researchers who prioritize accurate uncertainty quantification. By minimizing the $\chi$-divergence, it avoids the under-dispersion of KLVI and provides a tractable upper bound on model evidence. While the variance of the gradients remains a challenge for future research, the ability to "sandwich" the marginal likelihood represents a significant step forward in making variational methods more robust and interpretable.
Summary updated