Abstract
Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. However, SVGD has been found to suffer from variance underestimation when the dimensionality of the target distribution is high. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue, although this can severely overestimate the epistemic (model) uncertainty. In this work, we propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces. Compared with other variants of SVGD that rely on dimensionality reduction, GSVGD updates the projectors simultaneously for the score function and the data, and the optimal projectors are determined through a coupled Grassmann-valued diffusion process which explores favourable subspaces. Both our theoretical and experimental results suggest that GSVGD enjoys efficient state-space exploration in high-dimensional problems that have an intrinsic low-dimensional structure.
Introduction
Stein Variational Gradient Descent (SVGD) is a popular particle-based method for Bayesian inference that evolves a set of particles deterministically to match a target distribution. Despite its efficiency, SVGD faces a significant challenge in high-dimensional spaces: it often severely underestimates variance, a phenomenon known as "mode collapse." This occurs because the repulsive force between particles, intended to maintain diversity, weakens as dimensionality increases.
Previous attempts to mitigate this, such as Sliced-SVGD (S-SVGD), project data onto one-dimensional slices. While this addresses variance underestimation, 1D projections frequently fail to capture complex correlations and can lead to overestimating uncertainty. Grassmann Stein Variational Gradient Descent (GSVGD) provides a middle ground by projecting onto subspaces of arbitrary dimension $m$ and adaptively identifying the most informative projection angles.
Methodology
GSVGD introduces a novel approach to dimensionality reduction within the Stein framework:
- Grassmann Manifold Optimization: The algorithm searches for the $m$-dimensional subspace that maximizes the discrepancy between the current particle distribution and the target. This search is conducted on the Grassmann manifold—the space of all possible $m$-dimensional subspaces.
- Coupled ODE-SDE System: The particles evolve via a system of Ordinary Differential Equations (ODEs) to match the target distribution. Simultaneously, the projectors (subspaces) evolve through a Stochastic Differential Equation (SDE) on the Grassmann manifold. This SDE facilitates global exploration, using noise to avoid local minima.
- Unified Projection: Unlike sliced methods that treat the score function and data separately, GSVGD updates projectors for both simultaneously, ensuring particles are pushed toward the target in the most relevant directions.
Experimental Results
The authors evaluated GSVGD across synthetic benchmarks (Multivariate Gaussian, Multimodal Mixture, X-shaped Mixture) and real-world tasks (Conditioned Diffusion Process, Covertype Dataset).
Variance and Sample Quality
In a 100-dimensional Gaussian test, GSVGD demonstrated a "Goldilocks" effect. While SVGD's variance estimate collapsed toward zero and S-SVGD's estimate became inflated, GSVGD maintained a variance estimate near the true value of 1.0 across all dimensions.
On complex distributions like the X-shaped and Multimodal mixtures, GSVGD achieved lower Energy Distance—a metric for distributional matching—compared to both SVGD and S-SVGD. This suggests that even small increases in projection dimension (e.g., $m=2$ or $m=5$) significantly enhance the capture of variable interactions.
Real-World Application
On the Covertype dataset for Bayesian logistic regression, GSVGD's covariance matrix estimation was significantly closer to the Hamiltonian Monte Carlo (HMC) "gold standard" than its competitors, demonstrating its efficacy for real-world Bayesian posterior distributions.
Conclusion
GSVGD successfully addresses the curse of dimensionality in SVGD by leveraging adaptive subspace projections. By treating subspace selection as a diffusion process, the algorithm automatically discovers intrinsic low-dimensional structures. Future work will focus on automating hyperparameter tuning for the projection dimension $m$ and temperature $T$, as well as establishing formal convergence proofs for the coupled ODE-SDE system.
Summary updated