Introduction
Variational Inference (VI) is a cornerstone of modern Bayesian machine learning, used to approximate complex posterior distributions. However, standard Stochastic Gradient (SG) methods for VI often struggle with two main issues: ignoring the underlying geometry of probability distributions and the "conjugacy trap." Standard methods typically use Euclidean distance to determine step sizes, which is often a poor measure of dissimilarity for probability distributions. Furthermore, while "Natural Gradient" methods address geometry, they traditionally only work for "conditionally-conjugate" models. For non-conjugate models, such as Gaussian Processes with non-Gaussian likelihoods, these methods become computationally expensive or require complex approximations.
This paper introduces Proximal-Gradient Stochastic Variational Inference (PG-SVI), a framework that accelerates Bayesian inference by accounting for the geometry of probability distributions while maintaining flexibility for non-conjugate models.
The PG-SVI Framework
The core idea of PG-SVI is to split the Evidence Lower Bound (ELBO) objective into two parts:
- $f(\lambda)$: The "difficult" non-conjugate terms, which are approximated using a stochastic gradient.
- $h(\lambda)$: The "easy" conjugate terms, which are kept in their original form.
Instead of a standard gradient step, the authors use a proximal update that minimizes a linearized version of the difficult term plus the original easy term. Crucially, this update is constrained by a General Divergence Function $D$ rather than Euclidean distance. By choosing $D$ to be the Kullback-Leibler (KL) divergence, the algorithm naturally adapts to the information geometry of the parameter space. For many models, including Gaussian Processes, this "split-and-linearize" strategy results in closed-form updates, eliminating the need for expensive inner-loop optimizations.
Experimental Evaluation
The authors evaluated PG-SVI on Gaussian Process (GP) binary classification using the Sonar, Ionosphere, and USPS-3vs5 datasets, as well as Correlated Topic Modeling (CTM) on the NIPS and AP datasets.
In GP classification experiments, PG-SVI demonstrated remarkable efficiency, converging in fewer than 10 passes through the data. In contrast, standard gradient descent and popular adaptive methods like ADAM and RMSprop required over 100 passes to reach comparable performance. For CTM, PG-SVI achieved superior held-out log-likelihoods compared to traditional Mean-Field, Delta, and Laplace approximation methods. Additionally, for Gaussian Processes, the specific update rule allows for $O(N)$ memory complexity instead of the usual $O(N^2)$, significantly improving scalability.
Theoretical Insights and Robustness
This work provides the first formal convergence-rate analysis for VI methods using general divergence functions in non-convex settings. The authors prove that the method converges to a stationary point at a rate of $O(1/t)$.
A key insight is that using the correct divergence function allows for larger, more robust step sizes. As shown in the figure above, PG-SVI (red diamond) requires significantly fewer examples to converge across a wide range of step sizes compared to standard methods. The paper also proves that for a fixed accuracy, there is always a sufficiently small fixed step-size that will reach it, providing confidence for practitioners using constant learning rates.
Conclusion and Future Work
PG-SVI offers a powerful combination of geometric awareness and computational simplicity. By leveraging the mathematical structure of conjugate terms and using appropriate divergence functions, it achieves state-of-the-art convergence speeds. While the current work focuses heavily on Variational-Gaussian inference and requires some model-specific insight to define the objective "split," future work aims to extend these benefits to a broader class of non-conjugate exponential family models and automate the step-size tuning process.
Summary updated