In Bayesian inference, the difficulty of sampling is often attributed to high dimensionality. However, the true bottleneck is frequently not the number of parameters, but the geometry of the posterior distribution. When a distribution features varying scales, extreme curvature, or narrow "funnels," standard Markov Chain Monte Carlo (MCMC) algorithms struggle to explore the space, leading to slow convergence, high autocorrelation, and biased results.
The Geometry of Failure
Standard samplers like Hamiltonian Monte Carlo (HMC) or Langevin dynamics rely on a global step size. This becomes a liability in "pathological" landscapes where the ideal step size varies drastically across the space.
A classic example is the funnel geometry common in hierarchical Bayesian models. As Sutton et al. and Ghosh et al. describe, tight coupling between hyperparameters and lower-level parameters creates a region of high curvature (the "neck" of the funnel) and a region of low density (the "mouth"). A step size small enough to explore the neck will be too small to ever leave the mouth, while a large step size will cause the sampler to "crash" into the walls of the neck, leading to rejections.
Similar issues arise in Bayesian Inverse Problems, where solutions often concentrate along thin, winding ridges or manifolds Lan et al.. In these cases, the "stiffness" of the parameter space prevents local samplers from moving effectively without incorporating higher-order geometric information Beskos et al..
Adapting to Local Curvature
One way to overcome these bottlenecks is to make the sampler geometry-aware. Instead of a fixed Euclidean metric, researchers use Riemannian geometry to adapt the sampler's behavior to the local landscape.
Ma et al. provided a "complete recipe" for this, showing how any continuous-dynamic sampler can be parameterized to incorporate local curvature while maintaining correct convergence. By using a position-dependent metric tensor, the sampler effectively "flattens" the space, allowing for larger, more efficient steps even in regions of high curvature. To reduce the computational cost of these methods, Hartmann et al. introduced the Monge metric, which simplifies the required calculations from cubic to quadratic complexity.
Other approaches focus on specific random walks. Vempala et al. utilize the Dikin walk, an interior-point method that uses the Hessian of a barrier function to navigate "skinny" or ill-conditioned feasible regions. Similarly, Williams proposed a geodesic slice sampler that uses Riemannian paths to jump between isolated modes and navigate extreme curvature without the need for manual tempering.
Warping the Landscape with Transport Maps
Rather than changing how the sampler moves, an alternative strategy is to change the space itself. This is the intuition behind transport-based preconditioning and Normalizing Flows.
The goal is to learn a "transport map" that warps a complex, non-Gaussian posterior into a simple, isotropic Gaussian. Marzouk et al. demonstrated that even approximate maps built from low-fidelity models can act as powerful preconditioners, significantly speeding up sampling. Grenioux et al. further clarified that while global flow-based proposals help with multimodality, local reparameterization is often more robust for handling high-dimensional geometric pathologies.
This "warping" approach is particularly effective for hierarchical models. Moore showed that automatically switching between "centered" and "non-centered" reparameterizations can resolve the funnel pathologies that plague probabilistic programs. By transforming the model's structure, the sampler encounters a much smoother geometry, leading to faster convergence Arbel et al..
Learning the Geometry
Recent work has moved toward learning these geometric adaptations during the sampling process. Langosco di Langosco et al. introduced Neural Variational Gradient Descent (NVGD), which uses a neural network to learn optimal update directions, automatically adapting to narrow regions or ill-conditioned scales. Similarly, Schär et al. developed Parallel Affine Transformation Tuning (PATT), which uses multiple chains to learn a transformation that maps the target into an isotropic space, achieving high efficiency without requiring gradient information.
By treating the geometry of the posterior as a feature to be exploited rather than a hurdle to be cleared, these methods allow for reliable inference in models that were previously considered "un-sampleable."
Key Foundational Work
A Complete Recipe for Stochastic Gradient MCMC (2015) — contributed 1 concept elements to this direction.
Semi-Separable Hamiltonian Monte Carlo for Inference in Bayesian Hierarchical Models (2014) — contributed 1 concept elements to this direction.
Positive Curvature and Hamiltonian Monte Carlo (2014) — contributed 1 concept elements to this direction.