Abstract
Variational inference (VI) is widely used as an efficient alternative to Markov chain Monte Carlo. It posits a family of approximating distributions $q$ and finds the closest member to the exact posterior $p$. Closeness is usually measured via a divergence $D(q || p)$ from $q$ to $p$. While successful, this approach also has problems. Notably, it typically leads to underestimation of the posterior variance. In this paper we propose CHIVI, a black-box variational inference algorithm that minimizes $D_{\chi}(p || q)$, the $\chi$-divergence from $p$ to $q$. CHIVI minimizes an upper bound of the model evidence, which we term the $\chi$ upper bound (CUBO). Minimizing the CUBO leads to improved posterior uncertainty, and it can also be used with the classical VI lower bound (ELBO) to provide a sandwich estimate of the model evidence. We study CHIVI on three models: probit regression, Gaussian process classification, and a Cox process model of basketball plays. When compared to expectation propagation and classical VI, CHIVI produces better error rates and more accurate estimates of posterior variance.
Introduction
In Bayesian analysis, calculating the exact posterior distribution—the updated belief about model parameters after observing data—is often computationally intractable. Variational Inference (VI) addresses this by transforming the problem into optimization: selecting a simple distribution $q$ (e.g., a Gaussian) and adjusting its parameters to minimize its divergence from the true posterior $p$.
The standard approach, KLVI, minimizes the Kullback-Leibler (KL) divergence $D_{KL}(q||p)$. While efficient, KLVI is "zero-forcing," meaning it tends to fit the mode of the distribution while ignoring the tails. This leads to a systematic underestimation of posterior variance, making models overconfident. Furthermore, KLVI only provides a lower bound (ELBO) on the model evidence, leaving the gap to the true marginal likelihood unknown.
Methodology: CHIVI and the CUBO
To address these limitations, we introduce CHIVI, a black-box variational inference algorithm that minimizes the $\chi$-divergence ($D_\chi(p||q)$). Unlike the KL divergence used in standard VI, the $\chi$-divergence is "zero-avoiding." This property forces the approximation $q$ to cover all regions where the true posterior $p$ has significant mass, resulting in more realistic uncertainty estimates.
A key contribution of this work is the derivation of the CUBO ($\chi$ Upper Bound), a tractable upper bound on the log marginal likelihood. By combining the CUBO with the traditional ELBO, we obtain a "sandwich estimate" of the model evidence. As the order of the divergence decreases, these bounds tighten, providing a clear interval for the true marginal likelihood.
CHIVI is designed as a "black-box" algorithm, utilizing reparameterization gradients on the exponentiated CUBO. This ensures that stochastic gradients remain unbiased during optimization without requiring model-specific mathematical derivations.
Experimental Evaluation
We evaluated CHIVI across three distinct model types using several benchmark datasets:
- Bayesian Probit Regression: Tested on UCI datasets including Pima, Ionosphere, Madelon, and Covertype.
- Gaussian Process (GP) Classification: Evaluated on the Crabs, Sonar, and Ionosphere datasets.
- Cox Process Model: Applied to a spatial point process model of 2015-2016 NBA basketball shot data (over 150,000 shots from 308 players) to map shooting intensities and uncertainties.
Results and Discussion
Overcoming Overconfidence
Experimental results in Probit Regression and GP Classification show that CHIVI consistently yields lower classification error rates compared to standard black-box VI and Expectation Propagation (EP). Because the $\chi$-divergence is zero-avoiding, CHIVI produces "fatter" approximations that better represent true model uncertainty, particularly in light-tailed posteriors where standard VI often becomes unstable.
Realistic Uncertainty in Spatial Models
In the NBA basketball study, we compared CHIVI’s posterior variance estimates against ground truth calculated via Hamiltonian Monte Carlo (HMC). While standard VI (KLQP) was often overconfident in areas with sparse data, CHIVI correctly identified these regions as highly uncertain, mirroring the HMC results.
Global vs. Local Optimization
Unlike Expectation Propagation, which also captures variance well but can lack convergence guarantees, CHIVI optimizes a well-defined global objective. This ensures robust convergence while maintaining the benefits of capturing broader posterior mass.
Conclusion
CHIVI provides a principled framework for variational inference that avoids the overconfidence typical of KL-based methods. By providing an upper bound (CUBO) to complement the ELBO, it allows for more rigorous model evidence estimation. The primary challenge remains gradient variance; exponentiating the CUBO for unbiased optimization can lead to high-variance gradients. Future research will focus on variance reduction schemes specifically for expectations of likelihood ratios to enhance CHIVI's scalability for massive, complex datasets.
Summary updated