Riemannian Stein Variational Gradient Descent for Bayesian Inference

Chang Liu, Jun Zhu

Bayesian inference is a cornerstone of modern machine learning, yet standard methodologies often rely on the simplifying assumption that parameter spaces are Euclidean. While tools like Stein Variational Gradient Descent (SVGD) have proven effective, they struggle with models where parameters are naturally constrained—such as data on a hypersphere—or where the "distance" between distributions is better described by information geometry than by straight lines.

In their research, Liu and Zhu introduce RSVGD, a framework that generalizes particle-based inference to the curved surfaces of Riemannian manifolds. By incorporating the geometric curvature of the parameter space, RSVGD enables more accurate inference on manifolds and accelerates convergence in standard spaces by leveraging natural gradients.

The Challenge of Non-Euclidean Inference

Traditional Bayesian inference methods typically fall into three categories: Model-based Variational Inference (M-VI), which is fast but restricted by distributional assumptions; Monte Carlo (MC) methods, which are accurate but slow; and Particle-based Variational Inference (P-VI), which offers a middle ground. SVGD is a leading P-VI method, but its Euclidean bias leads to two primary issues:

  1. Geometric Constraints: Many models, such as directional data models, live on manifolds where Euclidean updates are invalid.
  2. Optimization Inefficiency: Even in flat spaces, ignoring the "information geometry" (the natural distance between probability distributions) can lead to suboptimal optimization paths.

Simply adapting manifold mathematics to SVGD is non-trivial due to challenges like coordinate variance and topological constraints (e.g., the Hairy Ball Theorem), which prevent the definition of smooth vector fields on certain manifolds.

The RSVGD Framework

The authors address these challenges by developing a rigorous mathematical foundation for RSVGD:

  • Riemannian Stein’s Identity: They derive a manifold-specific version of Stein’s identity, relating the target distribution to a Riemannian Stein operator.
  • Coordinate Invariance: RSVGD is designed to be invariant to the choice of local coordinate systems, ensuring consistent particle updates regardless of parameterization.
  • Functional Gradient in RKHS: By solving a maximization problem within a Reproducing Kernel Hilbert Space (RKHS) of vector fields, the authors derive a closed-form update rule for particles on the manifold.
  • Embedded Space Formulation: To handle manifolds like hyperspheres that are difficult to parameterize globally, they provide expressions for updates in the embedded Euclidean space, avoiding numerical instabilities at coordinate boundaries.

Experimental Evaluation

The effectiveness of RSVGD was evaluated across two distinct domains: Bayesian Logistic Regression in Euclidean space and a Spherical Admixture Model on a manifold.

Acceleration via Information Geometry

In Bayesian Logistic Regression tasks using the Splice19 and Covertype datasets, the authors used the Fisher Information metric to define the Riemannian manifold. This approach effectively implements a "natural gradient" update.

RSVGD demonstrates significantly faster convergence and higher test accuracy compared to standard SVGD on the Splice19 and Covertype datasets.

As shown in the results, RSVGD reaches higher test accuracy in significantly fewer iterations than standard SVGD, even when the latter is augmented with adaptive learning rates like AdaGrad. This highlights how geometry can act as an optimizer.

Inference on Manifolds

For the Spherical Admixture Model (SAM) applied to the 20 Newsgroups dataset, inference must be performed on a product manifold of hyperspheres—a task where standard SVGD is inapplicable.

On the Spherical Admixture Model, RSVGD achieves lower log-perplexity than Mean Field VI and various MCMC baselines, maintaining high efficiency as the number of particles increases.

RSVGD not only outperformed Mean Field Variational Inference but also proved more "iteration-effective" than Geodesic MCMC methods. While MCMC methods might eventually converge to similar results, RSVGD reached stable, high-quality solutions within 200 epochs, a timeframe where MCMC baselines were still struggling to converge.

Conclusion and Future Work

RSVGD represents a significant step forward for Bayesian inference, demonstrating that accounting for geometry—whether intrinsic to the model or derived from information theory—leads to more robust and efficient algorithms. While the computational cost of inverting the metric tensor and the noise sensitivity of mini-batching remain areas for improvement, RSVGD provides a mathematically grounded path for inference in complex, constrained parameter spaces. Future research may extend these principles to goodness-of-fit tests via Riemannian Kernelized Stein Discrepancy or apply them to deep generative modeling.

Summary updated

Abstract

We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifold. The benefits are two-folds: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and (ii) for inference tasks on Riemann manifolds, RSVGD brings the unique advantages of SVGD to the Riemannian world. To appropriately transfer to Riemann manifolds, we conceive novel and non-trivial techniques for RSVGD, which are required by the intrinsically different characteristics of general Riemann manifolds from Euclidean spaces. We also discover Riemannian Stein's Identity and Riemannian Kernelized Stein Discrepancy. Experimental results show the advantages over SVGD of exploring distribution geometry and the advantages of particle-efficiency, iteration-effectiveness and approximation flexibility over other inference methods on Riemann manifolds.
BibTeX
@inproceedings{liu2018riemannian,
  author = {Chang Liu and Jun Zhu},
  title = {{Riemannian Stein Variational Gradient Descent for Bayesian Inference}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=7gGsbXhvRGA},
}
Figure