Abstract
We develop Riemannian Stein Variational Gradient Descent (RSVGD), a Bayesian inference method that generalizes Stein Variational Gradient Descent (SVGD) to Riemann manifolds. The benefits are two-fold: (i) for inference tasks in Euclidean spaces, RSVGD has the advantage over SVGD of utilizing information geometry, and (ii) for inference tasks on Riemann manifolds, RSVGD brings the unique advantages of SVGD to the Riemannian world. To appropriately transfer to Riemann manifolds, we conceive novel and non-trivial techniques for RSVGD, which are required by the intrinsically different characteristics of general Riemann manifolds from Euclidean spaces. We also discover Riemannian Stein's Identity and Riemannian Kernelized Stein Discrepancy. Experimental results show the advantages over SVGD of exploring distribution geometry and the advantages of particle-efficiency, iteration-effectiveness and approximation flexibility over other inference methods on Riemann manifolds.
Introduction
Bayesian inference—the process of estimating a posterior distribution from data—is a cornerstone of machine learning. Traditional methods usually fall into two camps: Variational Inference (VI), which is fast but often inaccurate due to rigid structural assumptions, and Markov Chain Monte Carlo (MCMC), which is asymptotically accurate but slow to converge due to sample autocorrelation.
Stein Variational Gradient Descent (SVGD) recently emerged as a "best of both worlds" particle-based approach. However, SVGD is designed for flat Euclidean space. This limitation is significant because many models have parameters that naturally live on curved surfaces (e.g., directions on a hypersphere). Furthermore, even in flat space, the "information geometry" of the distribution itself can be used to speed up convergence, a factor standard SVGD ignores.
The RSVGD Framework
Applying SVGD to manifolds is not as simple as swapping coordinates; standard SVGD updates are not "coordinate invariant," meaning results could change depending on the local coordinate system used. The authors develop RSVGD by mathematically lifting the Stein operator into Riemannian geometry. Key technical contributions include:
- Riemannian Stein’s Identity & Discrepancy: New theoretical tools to measure the distance between distributions on a manifold.
- Coordinate Invariance: Ensuring the update rule (the functional gradient) is independent of the local coordinate system.
- The Beltrami-Laplace Operator: Utilizing this generalized operator instead of a simple gradient to account for the manifold's curvature.
- Embedded Space Formulation: A practical method for manifolds like hyperspheres that performs calculations in the surrounding Euclidean space while ensuring particles never leave the manifold surface, avoiding numerical instability.
Experimental Results
The authors evaluated RSVGD on benchmark datasets to test its effectiveness in both Euclidean and constrained manifold settings.
Bayesian Logistic Regression
On the Splice19 and Covertype datasets, RSVGD was tested in Euclidean space by treating the problem as a Riemannian one using the Fisher Information Metric. This allows the algorithm to navigate the posterior landscape more efficiently by "seeing" the curvature of the loss surface.
As shown in the results, RSVGD reached higher test accuracy in fewer iterations than standard SVGD, demonstrating the power of utilizing distribution geometry.
Spherical Admixture Model (SAM)
For inference on hyperspheres using the 20News-different dataset, RSVGD addressed the "Hairy Ball" problem—topological constraints that prevent smooth, non-zero vector fields on certain manifolds. RSVGD bypasses this by defining update directions through a specific Reproducing Kernel Hilbert Space (RKHS) of gradients.
In these tests, RSVGD significantly outperformed Geodesic Monte Carlo (GMC) and Variational Inference (VI). It achieved better log-perplexity much faster (iteration effectiveness) and with fewer particles (particle efficiency) than MCMC-based methods.
Conclusion
RSVGD generalizes particle-based inference to Riemannian manifolds, offering significant gains in efficiency and flexibility. By leveraging information geometry in Euclidean space and providing a robust framework for manifold-constrained tasks, it overcomes the limitations of standard SVGD. Future work will investigate the impact of mini-batch gradient noise on Riemannian dynamics and the selection of optimal kernels for more complex structures like Stiefel manifolds.
Summary updated