Introduction
Standard Variational Inference (VI) is a cornerstone of Bayesian machine learning, offering a computationally efficient way to approximate complex posterior distributions. However, it often underestimates uncertainty, leading to overconfident but inaccurate models. To address this, researchers introduce CHIVI ($\chi$-divergence Variational Inference), a black-box algorithm that minimizes the $\chi$-divergence to create an upper bound on model evidence. By combining this with the traditional lower bound, CHIVI allows researchers to "sandwich" the true model evidence and produce much more realistic estimates of posterior variance.
The Problem with Standard VI
In Bayesian modeling, finding the exact posterior distribution $p(z|x)$—how hidden variables $z$ behave given observed data $x$—is usually intractable. VI solves this by finding a simpler distribution $q$ that is "close" to $p$.
The industry standard, KLVI, minimizes the Kullback-Leibler (KL) divergence from $q$ to $p$. Unfortunately, KLVI exhibits "zero-forcing" behavior: it prefers to ignore parts of the true distribution rather than risk putting mass where the true distribution is zero. This results in "underdispersed" models that underestimate risk and variance. While alternatives like Expectation Propagation (EP) attempt to fix this, they often fail to converge and lack a clear global objective function.
CHIVI and the CUBO Method
Instead of the standard KL divergence, CHIVI minimizes the $\chi^2$-divergence from the posterior $p$ to the approximation $q$. This shift in divergence metric yields several powerful technical contributions:
- The CUBO (Chi Upper Bound): The authors derive a new objective function that serves as a rigorous upper bound on the log marginal likelihood ($\log p(x)$).
- Zero-Avoiding Behavior: Unlike KL divergence, the $\chi$-divergence is "zero-avoiding." It becomes infinite if $q(z)$ is zero while $p(z|x)$ is positive. This forces the approximation $q$ to cover the entire support of the posterior, preventing the overconfidence seen in standard VI.
- Black-Box Optimization: CHIVI does not require complex, model-specific derivations. It uses reparameterization gradients on the exponentiated CUBO to ensure estimates remain unbiased during optimization.
The Sandwich Theorem and Key Results
By utilizing the standard ELBO (a lower bound) alongside the new CUBO (an upper bound), researchers can bracket the true model evidence. This "sandwich estimate" is highly valuable for model selection and evaluating the tightness of the approximation.
The authors evaluated CHIVI across Bayesian Probit Regression, Gaussian Process (GP) Classification, and Cox Processes. The results highlight CHIVI's strengths:
- Tight Evidence Bracketing: As optimization progresses, the gap between the ELBO and CUBO narrows, providing a reliable, bounded range for the true model evidence.
- Lower Error Rates: On GP classification tasks—where standard VI often struggles due to light-tailed posteriors—CHIVI outperformed both the Laplace approximation and EP on benchmark datasets like Sonar and Ionosphere.
- Better Uncertainty Estimation: To test spatial modeling, the authors applied a Cox process to NBA shot patterns from the 2015-2016 season. CHIVI produced posterior uncertainty estimates that closely matched Hamiltonian Monte Carlo (HMC, the gold standard), whereas standard VI significantly underestimated uncertainty in sparse data regions.
Limitations and Future Work
The primary drawback of CHIVI is computational variance. Exponentiating the CUBO bound to obtain unbiased gradients can lead to high variance during optimization. Developing more sophisticated variance reduction schemes for these types of objectives remains a critical open research problem. Additionally, while CHIVI excels at capturing variance and avoiding zero-forcing, it may be less suited for multimodal posteriors where focusing on a single mode is preferred for specific predictive tasks.
Summary updated