Abstract
We present a selection criterion for the Euclidean metric adapted during warmup in a Hamiltonian Monte Carlo sampler that makes it possible for a sampler to automatically pick the metric based on the model and the availability of warmup draws. Additionally, we present a new adaptation inspired by the selection criterion that requires significantly fewer warmup draws to be effective. The effectiveness of the selection criterion and adaptation are demonstrated on a number of applied problems. An implementation for the Stan probabilistic programming language is provided.
Introduction
Hamiltonian Monte Carlo (HMC) has become a cornerstone of modern Bayesian inference, particularly through its implementation in probabilistic programming languages like Stan. However, its efficiency is highly sensitive to the choice of the "Euclidean metric" (or mass matrix), which acts as a preconditioner for the posterior distribution. A well-chosen metric can transform a highly correlated or poorly scaled posterior into a more isotropic space, allowing for larger integration timesteps and more efficient exploration.
Historically, practitioners have faced a binary choice: simple diagonal metrics, which are easy to estimate but ignore parameter correlations, or dense metrics, which capture correlations but require a prohibitive number of warmup samples to estimate reliably. This paper introduces a middle ground: an automated selection criterion to choose the optimal metric type and a low-rank Hessian-based adaptation method that works effectively even with limited warmup data.
Methodology
The Selection Criterion
The authors derive a selection criterion based on the stability limits of the leapfrog integrator. The speed of HMC is fundamentally limited by the direction of highest curvature, which dictates the maximum allowable timestep. The proposed criterion predicts the Effective Sample Size per second (ESS/s) by measuring the "condition number" of the linearized dynamics. This allows the sampler to evaluate, at the end of each warmup window, whether switching to a more complex metric (e.g., from diagonal to dense) is likely to yield a net performance gain.
Low-Rank Hessian Adaptation
To address the "chicken and egg" problem of needing good samples to estimate a dense metric, the authors propose using local curvature information. By employing the Lanczos algorithm, the sampler can identify the directions of highest curvature (the largest eigenvalues of the Hessian) using only a few gradient evaluations.
As shown in Figure 2, the raw eigenvalue spectrum (circles) often spans several orders of magnitude, creating a bottleneck for the integrator. The proposed adaptation (pluses) effectively compresses this spectrum, significantly improving the condition number of the problem and allowing for larger timesteps.
Hybrid Regularization
The method combines this local Hessian information with global sample covariance estimates using an inverse-Wishart prior. This hybrid approach allows the sampler to start with a "smart guess" based on local geometry and transition to a robust global estimate as more warmup draws are collected.
Key Insights: Timestep Bottlenecks and Geometry
The primary bottleneck in HMC is often a single "tight" direction in the parameter space. By identifying these directions via the Hessian, the sampler can "stretch" them out, enabling much larger timesteps.
However, this approach relies on the assumption that local curvature is a reasonable proxy for global behavior. Figure 1 illustrates a typical log-density profile; while the curvature provides a good local approximation, the sampler must still account for non-linearities as it moves through the posterior's "typical set."
Experimental Results
The proposed methods were benchmarked across several challenging models:
- Kilpisjärvi: A linear regression with extreme correlation. The Hessian-based metric achieved an ESS/s of ~22,000–33,000, a massive improvement over the ~110–150 achieved by standard diagonal metrics.
- Diamonds & Prophet: In these models, the low-rank adaptation reached high efficiency much earlier in the warmup phase than traditional dense metrics.
- Radon: In this high-dimensional hierarchical model (389 parameters), the selection criterion correctly identified that the overhead of dense matrix operations outweighed the benefits, successfully defaulting to a diagonal metric to maintain overall efficiency.
Conclusion
This work provides a robust framework for automating metric selection in HMC. By leveraging local Hessian information and a principled selection criterion, samplers can achieve significant speedups on models with complex correlations while maintaining robustness on high-dimensional problems where dense metrics are impractical. Future work may explore extending this framework to more non-linear geometries where curvature varies more drastically across the posterior.
Summary updated