Abstract
We present a novel method for approximate inference. Using some of the constructs from expectation propagation (EP), we derive a lower bound of the marginal likelihood in a similar fashion to variational Bayes (VB). This hybrid approach, Tilted Variational Bayes (TVB), provides a mathematically stable objective function that remains robust even for light-tailed likelihoods where traditional VB fails.
Introduction
Approximate inference is a cornerstone of modern Bayesian machine learning, yet practitioners often face a trade-off between the mathematical stability of Variational Bayes (VB) and the flexibility of Expectation Propagation (EP). VB treats inference as an optimization problem, providing a rigorous lower bound on the evidence (ELBO). However, it struggles with "light-tailed" likelihoods common in classification—such as step functions—where the Kullback-Leibler (KL) divergence can become infinite, causing the method to fail.
Conversely, EP handles these likelihoods gracefully and is considered the gold standard for Gaussian Process classification. However, EP is notoriously difficult to implement, lacks global convergence guarantees, and does not provide a clean lower bound for hyperparameter optimization. Tilted Variational Bayes (TVB) aims to bridge this gap by marrying the stability of VB with the likelihood-handling capabilities of EP.
The Tilted Distribution Framework
The core innovation of TVB is the construction of the "Tilted Distribution" $q(x)$. Unlike standard VB, which typically uses a simple parametric family like a Gaussian, TVB incorporates the likelihood term directly into the approximation.
This framework uses "pseudo-data" parameters—a concept borrowed from EP—to represent the influence of the data. By including the likelihood inside the approximation, TVB ensures that the tails of the approximation match the tails of the true posterior. This prevents the "infinite KL" failure mode and allows for the derivation of a new, well-behaved lower bound on the marginal likelihood.
Because TVB provides a valid lower bound, the entire inference problem can be solved using standard gradient-based optimizers like L-BFGS. This simplifies implementation significantly, turning a complex message-passing problem into a straightforward optimization of both the approximation and the model hyperparameters simultaneously.
Experimental Evaluation
The authors evaluated TVB across several benchmarks, including 12 UCI classification datasets and robust regression tasks.
Gaussian Process Classification: On standard benchmarks like Banana and German Credit, TVB achieved predictive performance nearly identical to EP in terms of accuracy and hold-out likelihood. In some instances, such as the Heart dataset, TVB provided even better hyperparameter estimates than EP due to its more reliable objective function.
Hyperparameter Optimization: A key advantage of TVB is its ability to provide a smooth surface for hyperparameter tuning. Using the Ionosphere dataset, the authors demonstrated how the marginal likelihood surface can be effectively navigated to find optimal lengthscales and signal variances.
Robustness to Outliers: In a robust GP regression task using historical Olympic marathon data, TVB successfully handled the 1904 outlier—a race run in extreme heat where many athletes quit. While standard GPs were skewed by this outlier, TVB (using a Student-t likelihood) correctly captured the true trend of improving athletic performance over time.
Discussion and Conclusion
Tilted Variational Bayes represents a significant advancement in approximate inference. It offers the robustness of EP with the mathematical rigor and ease of implementation of VB. By providing a guaranteed lower bound, TVB serves as a more reliable objective function for automated machine learning pipelines.
While the current implementation focuses on unimodal posteriors and has been tested on small-to-medium datasets, the potential for parallelization and stochastic optimization suggests a clear path toward scaling TVB for large-scale Bayesian modeling. TVB effectively turns a complex inference challenge into a robust, scalable optimization problem.
Summary updated