Abstract
The Hamiltonian Monte Carlo method generates samples by introducing a mechanical system that explores the target density. For distributions on manifolds it is not always simple to perform the mechanics as a result of the lack of global coordinates, the constraints of the manifold, and the requirement to compute the geodesic flow. In this paper we explain how to construct the Hamiltonian system on naturally reductive homogeneous spaces using symplectic reduction, which lifts the HMC scheme to a matrix Lie group with global coordinates and constant metric. This provides a general framework that is applicable to many manifolds that arise in applications, such as hyperspheres, hyperbolic spaces, symmetric positive-definite matrices, Grassmannian, and Stiefel manifolds.
Introduction
Sampling from probability distributions defined on non-Euclidean spaces is a recurring challenge in modern statistics and machine learning. Manifolds such as spheres, rotation groups, and spaces of positive-definite matrices appear frequently in directional statistics, robotics, and covariance estimation. While standard Hamiltonian Monte Carlo (HMC) is highly effective in flat Euclidean space ($\mathbb{R}^n$), extending it to curved manifolds introduces significant hurdles:
- Coordinate Singularities: Local charts (e.g., latitude and longitude) inevitably possess singularities like poles where numerical integration fails.
- Constraint Enforcement: Embedding a manifold in a higher-dimensional space requires expensive projection steps or Lagrange multipliers to ensure the sampler remains on the surface.
- Complex Geodesics: Computing the "straightest" path on a curved surface (the geodesic flow) is often computationally prohibitive and lacks simple closed-form expressions.
Methodology: Lifting and Reduction
The core contribution of this work is a framework that bypasses these issues by "lifting" the sampling problem from the manifold $\mathcal{M}$ to a related matrix Lie group $\mathcal{G}$.
Lifting to Lie Groups
By representing the manifold as a quotient space $\mathcal{M} = \mathcal{G}/\mathcal{K}$, where $\mathcal{G}$ is a Lie group and $\mathcal{K}$ is a subgroup, the authors utilize a global matrix representation. This eliminates the need for local coordinate charts and their associated singularities.
Symplectic Reduction
The authors employ symplectic reduction to simplify the dynamics. By initializing the system on the Lie group with specific momentum constraints (setting certain components to zero), the physical laws of the system naturally restrict the trajectory to the manifold. This "automatic" constraint enforcement leverages Noether's Theorem, ensuring the sampler stays on $\mathcal{M}$ without explicit projection steps.
Matrix Integrators
To maintain the geometric properties of the Hamiltonian system, the paper introduces specialized numerical integrators. These integrators use matrix exponentials to compute geodesic flows, ensuring the mapping remains "symplectic"—preserving the underlying volume and energy of the physical system. On a Lie group with a left-invariant metric, the velocity update is simplified because gradients only need to be computed at the identity.
Validation and Key Results
The framework was validated on several synthetic benchmarks, including the 2-sphere ($S_2$) and hyperbolic spaces ($H_2^1, H_2^2$).
Numerical Stability and Scaling
A critical requirement for HMC is the conservation of the Hamiltonian (energy). The proposed symplectic integrators demonstrate exceptional stability, with energy errors ($\delta H$) remaining small and stable even over long trajectories. Furthermore, the error scales predictably as $\delta t^2$ for the standard Leapfrog integrator, confirming that the "lifted" dynamics do not introduce unexpected numerical artifacts.
Global Sampling Performance
The method's ability to navigate complex potentials is demonstrated in Figure 1. The sampler successfully explores a distribution forming the letters "HMC" on a sphere, traversing poles and high-curvature regions without slowing down or losing accuracy.
Discussion and Limitations
While the framework provides a unified recipe for sampling on "naturally reductive homogeneous spaces," it has some limitations:
- Computational Cost: Matrix operations, specifically matrix exponentiation, scale as $O(n^3)$, which may become a bottleneck for very high-dimensional manifolds.
- Tuning: Like standard HMC, parameters such as step size and trajectory length require tuning. Integration with adaptive methods like the No-U-Turn Sampler (NUTS) is a promising direction for future work.
- Scope: The method is tailored for spaces with specific symmetries (homogeneous spaces) and may not directly apply to arbitrary, unstructured manifolds.
In conclusion, this symplectic reduction approach offers a robust, mathematically principled way to perform HMC on a wide variety of manifolds used in modern data science, providing global coordinates and stable numerical integration.
Summary updated