Grassmann Stein Variational Gradient Descent

Xing Liu, Harrison Zhu, Jean-François Ton, George Wynne, Andrew B. Duncan

Abstract: Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. However, SVGD has been found to suffer from variance underestimation when the dimensionality of the target distribution is high. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue, although this can severely overestimate the epistemic (model) uncertainty. In this work, we propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces. Compared with other variants of SVGD that rely on dimensionality reduction, GSVGD updates the projectors simultaneously for the score function and the data, and the optimal projectors are determined through a coupled Grassmann-valued diffusion process which explores favourable subspaces. Both our theoretical and experimental results suggest that GSVGD enjoys efficient state-space exploration in high-dimensional problems that have an intrinsic low-dimensional structure.

Introduction

Stein Variational Gradient Descent (SVGD) has emerged as a powerful tool for approximating complex probability distributions by iteratively moving a set of "particles" toward high-probability regions. Despite its popularity, SVGD faces a significant challenge in high-dimensional spaces: the "repulsive" force intended to maintain particle diversity often vanishes. This phenomenon leads to variance underestimation, where the resulting particle set is overly concentrated, failing to represent the true uncertainty of the target distribution.

Previous attempts to mitigate this, such as Sliced-SVGD (S-SVGD), utilize 1D projections to simplify the problem. While S-SVGD addresses the collapse issue, 1D projections frequently fail to capture complex correlations between dimensions, often resulting in inflated variance and overestimation of uncertainty. Furthermore, S-SVGD typically employs fixed directions for the score function, which may not align with the most informative features of the data.

Methodology

To overcome these limitations, we introduce Grassmann Stein Variational Gradient Descent (GSVGD). This method generalizes the projection approach from 1D lines to $m$-dimensional subspaces, optimized dynamically on the Grassmann manifold.

The Grassmann Manifold and GKSD

Instead of selecting random directions, GSVGD treats subspace selection as an optimization problem on the Grassmann manifold—the mathematical space of all possible $m$-dimensional subspaces. We define the Grassmann Kernel Stein Discrepancy (GKSD) to identify the "worst-case" subspace where the current particle distribution deviates most significantly from the target.

Coupled ODE-SDE System

GSVGD evolves particles and projectors simultaneously using a coupled dynamical system:

  1. Particles follow an Ordinary Differential Equation (ODE) to move toward the target density.
  2. Projectors (subspaces) follow a Stochastic Differential Equation (SDE) on the Grassmann manifold.

This SDE acts as a simulated annealing process, allowing the algorithm to dynamically "hunt" for informative subspaces and avoid local minima. By using multiple projectors ($M$) in parallel, the algorithm can update particles along several critical subspaces in each step, ensuring robust exploration of the state space.

Experimental Evaluation

We evaluated GSVGD against standard SVGD and S-SVGD across synthetic toy examples, conditioned diffusion processes, and real-world Bayesian logistic regression tasks.

Variance Accuracy in High Dimensions

In high-dimensional Gaussian tests (up to 100D), standard SVGD's variance collapses toward zero as dimensionality increases. Conversely, S-SVGD tends to overestimate the variance. GSVGD, particularly with $m=1$ or $m=2$, maintains a variance remarkably close to the true value of 1.0 across all tested dimensions.

Variance of particles across dimensions. SVGD underestimates variance, while S-SVGD overestimates it. GSVGD variants maintain near-optimal variance.

Multimodal and Correlated Distributions

On multimodal mixtures and X-shaped distributions, GSVGD achieves significantly lower energy distance compared to its predecessors. By utilizing $m > 1$ dimensions, GSVGD captures internal correlations that 1D methods miss, providing a more faithful representation of the target geometry.

Comparison of particle distributions on a multimodal target. SVGD suffers from mode collapse, whereas GSVGD captures the target distribution more effectively.

Real-World Application: Covertype Dataset

In Bayesian logistic regression on the Covertype dataset (54 features), GSVGD produced covariance matrices that closely matched the gold-standard Hamiltonian Monte Carlo (HMC) results. In contrast, SVGD severely underestimated correlations, while S-SVGD significantly overestimated them.

Discussion and Future Work

GSVGD demonstrates that projecting onto $m$-dimensional subspaces provides a critical balance between the computational efficiency of dimensionality reduction and the representational power needed for high-dimensional inference. The use of a diffusion process on the Grassmann manifold ensures that the algorithm adaptively finds the most relevant subspaces as the particles evolve.

While GSVGD introduces hyperparameters such as the projection dimension ($m$) and the number of projectors ($M$), it offers a more stable and accurate alternative for high-dimensional problems with intrinsic low-dimensional structures. Future research may focus on automating hyperparameter selection through adaptive schemes or investigating the sensitivity of the SDE's temperature parameter.

Summary updated

Abstract

Stein variational gradient descent (SVGD) is a deterministic particle inference algorithm that provides an efficient alternative to Markov chain Monte Carlo. However, SVGD has been found to suffer from variance underestimation when the dimensionality of the target distribution is high. Recent developments have advocated projecting both the score function and the data onto real lines to sidestep this issue, although this can severely overestimate the epistemic (model) uncertainty. In this work, we propose Grassmann Stein variational gradient descent (GSVGD) as an alternative approach, which permits projections onto arbitrary dimensional subspaces. Compared with other variants of SVGD that rely on dimensionality reduction, GSVGD updates the projectors simultaneously for the score function and the data, and the optimal projectors are determined through a coupled Grassmann-valued diffusion process which explores favourable subspaces. Both our theoretical and experimental results suggest that GSVGD enjoys efficient state-space exploration in high-dimensional problems that have an intrinsic low-dimensional structure.
BibTeX
@inproceedings{liu2022grassmann,
  author = {Xing Liu and Harrison Zhu and Jean-François Ton and George Wynne and Andrew B. Duncan},
  title = {{Grassmann Stein Variational Gradient Descent}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=_ZWIbATvLE},
}
Figure