Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions

Mohammad Emtiyaz Khan, Reza Babanezhad, Lin Wu, Mark Schmidt, Masashi Sugiyama

Abstract

Several recent works have explored stochastic gradient methods for variational inference that exploit the geometry of the variational-parameter space. However, the theoretical properties of these methods are not well-understood and these methods typically only apply to conditionally-conjugate models. We present a new stochastic method for variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-form updates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space. Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectives by using a more general class of divergence functions. Beyond giving a theoretical justification for a variety of recent methods, our experiments show that new algorithms derived in this framework lead to state of the art results on a variety of problems. Further, due to its generality, we expect that our theoretical analysis could also apply to other applications.

Introduction: The Challenge of Geometry and Non-Conjugacy

Variational Inference (VI) has become a cornerstone of Bayesian machine learning, transforming the complex task of posterior estimation into a more manageable optimization problem. To handle modern "big data" scales, researchers frequently turn to Stochastic Variational Inference (SVI). However, standard SVI implementations often suffer from two significant limitations.

First is Geometry Blindness. Standard stochastic gradient methods typically rely on Euclidean distance to measure parameter updates. In the context of probability distributions, Euclidean distance is often a poor metric; a small numerical change in parameters can lead to a massive shift in the resulting distribution, or conversely, a large parameter change might have negligible impact on the distribution. Second is the Conjugacy Trap. Methods designed to account for this geometry, such as natural gradient descent, are generally restricted to "conditionally-conjugate" models. When faced with "non-conjugate" models—like Gaussian Processes with non-Gaussian noise—these elegant updates fail, forcing a return to slower, generic numerical optimization techniques.

The PG-SVI Framework

To address these hurdles, this paper introduces Proximal-Gradient Stochastic Variational Inference (PG-SVI). This framework unifies several optimization concepts to provide both scalability and geometric awareness:

  • Objective Splitting: The Evidence Lower Bound (ELBO) is split into a "difficult" term (typically the likelihood) and an "easy" term (the prior or entropy).
  • Linearization: The difficult term is linearized using a stochastic gradient at each step. This allows the framework to handle non-conjugate terms without requiring expensive inner-loop numerical solvers.
  • General Divergence Functions: Instead of Euclidean distance, PG-SVI utilizes a general class of divergence functions (such as KL-divergence or Bregman divergences) to define the "proximal" step. This ensures that parameter updates respect the natural geometry of the probability space.
  • Closed-Form Updates: By strategically choosing the split and divergence function, the authors derive simple, closed-form updates for complex models like Gaussian Processes (GPs) and Correlated Topic Models (CTMs), even in non-conjugate settings.

Key Results

The authors evaluated PG-SVI on standard benchmarks, including GP classification tasks (Sonar, Ionosphere, and USPS-3vs5) and Correlated Topic Models (NIPS and AP datasets).

Performance comparison of PG-SVI against state-of-the-art stochastic optimizers on three GP classification benchmarks.

In GP classification experiments, PG-SVI demonstrated a massive speedup in data passes. While standard SGD and popular adaptive methods like ADAM or ADAGRAD often required over 100 passes to reach convergence, PG-SVI consistently achieved superior performance within just 10 passes. Furthermore, for Correlated Topic Models, PG-SVI achieved better held-out log-likelihoods than established methods like the Delta method or Laplace approximation, as it avoids the crude Taylor expansions those methods rely on.

Theoretical and Empirical Insights

A major contribution of this work is the first formal convergence analysis for geometry-aware methods in non-convex settings. The authors established an $O(1/t)$ convergence rate to a stationary point for both deterministic and stochastic versions of the algorithm.

Analysis of convergence speed relative to step size for different optimization frameworks.

Empirically, the paper validates that using an appropriate divergence function allows for larger step sizes and faster convergence rates than standard gradient descent. By isolating "difficult" terms and linearizing them, PG-SVI maintains the efficient "natural" updates characteristic of conjugate models even when the model itself is non-conjugate.

Conclusion and Future Work

PG-SVI bridges the gap between the scalability of stochastic gradients and the efficiency of geometry-aware optimization. While the framework currently requires some manual step-size tuning, its ability to provide closed-form updates for non-conjugate models represents a significant leap forward. Future work may explore automated step-size rules and the extension of this framework to a broader class of non-conjugate exponential family models.

Summary updated

Abstract

Several recent works have explored stochastic gradient methods for variational inference that exploit the geometry of the variational-parameter space. However, the theoretical properties of these methods are not well-understood and these methods typically only apply to conditionally-conjugate models. We present a new stochastic method for variational inference which exploits the geometry of the variational-parameter space and also yields simple closed-form updates even for non-conjugate models. We also give a convergence-rate analysis of our method and many other previous methods which exploit the geometry of the space. Our analysis generalizes existing convergence results for stochastic mirror-descent on non-convex objectives by using a more general class of divergence functions. Beyond giving a theoretical justification for a variety of recent methods, our experiments show that new algorithms derived in this framework lead to state of the art results on a variety of problems. Further, due to its generality, we expect that our theoretical analysis could also apply to other applications.
BibTeX
@inproceedings{khan2016faster,
  author = {Mohammad Emtiyaz Khan and Reza Babanezhad and Lin Wu and Mark Schmidt and Masashi Sugiyama},
  title = {{Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2016},
  url = {https://openreview.net/forum?id=M-uWB5Fgpx},
}
Figure