Reading Curvature from Oscillations in Alternating Direction Hamiltonian Monte Carlo

Hamiltonian Monte Carlo (HMC) is a sampling method that treats the negative log-density as a landscape and moves a particle across it by simulating frictionless physics. In a narrow valley the particle overshoots and bounces between the walls. The optimization literature treats this bouncing as a pathology to be damped or reset. We propose to treat it as a free measurement instead. When HMC's standard integrator, leapfrog, runs on a locally quadratic region, every curvature direction turns into a rotation at a fixed angle per step. A short buffer of recent positions therefore satisfies a low-order linear recurrence, and the roots of that recurrence give the local curvatures and their directions. Recovering them needs no Hessian and no extra gradient evaluations.

The place where this measurement is most needed is Alternating Direction HMC (AD-HMC). This recent sampler alternates forward and backward Hamiltonian motion so that the method stays valid even when the momentum distribution is non-Gaussian and asymmetric. Its authors use this freedom to fit mode-specific momentum distributions to multimodal targets, but they note that the required local Hessians are computationally prohibitive. Our claim is that the oscillations AD-HMC already produces during warmup contain enough curvature information to build those mode-matched momentum distributions, at a small fraction of the Hessian cost.

Core Claim

Consider a quadratic potential with Hessian eigenvalue $\lambda_i$ along direction $v_i$, and leapfrog step size $\epsilon$. The linearized one-step leapfrog map rotates the $(q, p)$ pair along $v_i$ by an angle $\theta_i$ satisfying

$$\cos\theta_i = 1 - \tfrac{1}{2}\epsilon^2\lambda_i ,$$

and stability requires $\epsilon^2\lambda_i < 4$. The sequence of positions $q_t$ along a trajectory therefore obeys a linear recurrence whose characteristic roots are $e^{\pm i\theta_i}$. Fitting that recurrence to a buffer of $k$ positions returns every $\theta_i$ the buffer excites, and inverting the display equation gives $\lambda_i$. The residual vectors of the fit give the directions $v_i$.

The stiffest directions have the largest $\theta_i$. They are the ones that limit the step size and cause rejections, and they are also the ones that show up clearly in short buffers. The probe is therefore most accurate exactly where preconditioning pays off most.

We claim that a warmup procedure built on this probe can match Hessian-based mode adaptation in AD-HMC on sampling quality, at far lower cost, and can beat standard sample-covariance mass-matrix adaptation on targets whose modes have different local geometry. Adaptation runs only during warmup and the resulting momentum distribution is then frozen, so AD-HMC's correctness guarantees carry over unchanged.

Relation to Prior Work

The method we build on is Hamiltonian Monte Carlo with asymmetrical momentum distributions. That paper introduces AD-HMC, proves geometric convergence in Wasserstein distance under weaker conditions than symmetric HMC, extends the analysis to leapfrog with a Metropolis-Hastings correction, and proposes mode-matched auxiliary distributions whose Hessian cost it flags as a limitation. We keep its sampler and its guarantees and replace only the curvature source. Related operator-theoretic work by the same group, On L^q convergence of the Hamiltonian Monte Carlo, explains why self-adjointness matters. It is the reason we insist on warmup-only adaptation rather than state-dependent adaptation during sampling.

The extraction technique comes from the momentum-oscillation literature. Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration shows that alternating-direction optimizers have complex-eigenvalue transition maps, so their iterates spiral. It uses minimal polynomial extrapolation over a history buffer, at $O(nq^2)$ overhead, to jump to the spiral's center. We use the same recurrence fit for a different purpose: we read off the eigenvalues rather than extrapolating to a fixed point. The leapfrog setting is also better posed than ADMM's, because the roots lie on the unit circle rather than inside it.

Several papers supply the idea that oscillation is information rather than only noise, while constraining how it can be used in sampling. A Differential Equation for Modeling Nesterov's Accelerated Gradient Method and Practical Perspectives on Symplectic Accelerated Optimization kill oscillations with speed and gradient restarts. The second paper reports iteration counts dropping "from over 15,000 to just 51". Restarts, however, work by removing kinetic energy, which is precisely what a sampler must conserve. This is the main negative constraint that pushed us away from importing restarts into HMC.

Gradient-adjusted underdamped Langevin dynamics for sampling takes a different route. It adds Hessian-driven damping to underdamped Langevin dynamics and gets $\sqrt{\kappa}$ condition-number dependence, but it relies on Euler-Maruyama discretization with no accept/reject step. It changes the dynamics; we change only the metric the dynamics run in.

Finally, The No-U-turn sampler and its convergence analysis On the convergence of dynamic implementations of Hamiltonian Monte Carlo and No U-Turn Samplers handle a different symptom of momentum: looping back toward the start. They choose how long to integrate, not what geometry to integrate in, so they combine with our approach rather than compete with it.

Research Program

The first component is the estimator. During each warmup trajectory, forward or backward, we keep a rolling buffer of $k$ positions (around 8 to 32). We project the buffer onto the subspace spanned by its own differences and fit a recurrence of order up to $2r$ by least squares in the reduced-rank-extrapolation style. We then extract $r$ angles, the corresponding curvatures, and their directions.

The fit residual serves as a quadraticity check. When the local landscape is far from quadratic, the residual is large and the estimate is discarded. This matters on funnels and near boundaries, where the linear-rotation model is wrong.

The alternating structure of AD-HMC gives each region two independent trajectory segments with opposite time direction. We use their agreement as a second consistency check, which a single-direction sampler does not provide.

The second component is aggregation. Warmup samples are assigned to modes by a cheap clustering step. Within each mode, accepted estimates are pooled into a low-rank-plus-diagonal curvature model, formed from the stiff directions measured by oscillation plus a diagonal filled in from sample variances. This model sets the covariance and center of that mode's component in AD-HMC's asymmetric momentum mixture.

The third component is analysis. For Gaussian targets we will bound the estimation error of $\lambda_i$ as a function of buffer length and step size. The recurrence identity is exact there, so the error comes only from finite buffer length and floating-point conditioning. We will also characterize which low-curvature directions are missed because a short buffer never completes enough of a slow rotation. Those directions are the honest limit of the method, and the diagonal fallback exists to cover them.

Baselines

The naive reference point is AD-HMC with an identity or fixed isotropic momentum distribution. It is run with the same total gradient budget and a dual-averaging step size, as in The No-U-turn sampler. Any adaptation should beat it clearly on ill-conditioned targets. If it does not, the targets are too easy.

The strongest direct competitor is AD-HMC with the Hessian-based mode adaptation from Hamiltonian Monte Carlo with asymmetrical momentum distributions. No reported result found on Lacuna for its effective sample size on the benchmarks below, so we will rerun it ourselves. Hessians will be computed as exact Hessian-vector-product Lanczos solves, with their gradient-equivalent cost counted. Beating it means matching its effective sample size per sampling-phase gradient within noise while spending at least five times less warmup compute.

The strongest standard practice is NUTS with windowed diagonal and dense sample-covariance mass-matrix adaptation, given an identical warmup length. The reference number from the file is that NUTS outperforms the best hand-tuned HMC by a "factor of three" in effective sample size per gradient on a 250-dimensional correlated Gaussian and on a stochastic volatility model (The No-U-turn sampler). Beating it means higher minimum effective sample size per gradient across the full run, including warmup, on multimodal targets whose modes differ in curvature. On unimodal targets we expect only parity, and we will report it as such.

For Bayesian neural network regression we also compare against gradient-adjusted underdamped Langevin. It reports a validation RMSE of 5.90 ± 0.14 on Concrete Compressive Strength, against 6.28 ± 0.24 for plain underdamped Langevin (Gradient-adjusted underdamped Langevin dynamics for sampling). We will use the same single-hidden-layer ReLU network and splits. Beating it means lower RMSE at equal gradient count.

Two ablations isolate the key ingredient. The first swaps the oscillation probe for random directions, with curvatures estimated by finite differences at matched cost. It tests whether reading the oscillations adds anything beyond having some low-rank model. The second runs the same probe inside standard symmetric HMC with a single global metric. It tests whether AD-HMC's per-mode asymmetric momentum is what converts the curvature estimates into gains on multimodal targets.

Why not just…?

Why not just use NUTS, which already stops momentum from looping back? NUTS picks a path length and leaves the integration geometry alone. In a stiff valley it still takes tiny steps set by the largest curvature. It also relies on symmetric Gaussian momentum, so it cannot host per-mode asymmetric components. Non-Volume Preserving Hamiltonian Monte Carlo and No-U-TurnSamplers further reports standard NUTS failing to produce 2,000 samples in 24 hours on a 20-dimensional discontinuous target. The fair test is to give NUTS our learned metric as well. If NUTS with our metric beats AD-HMC with our metric, then the contribution is the probe rather than the sampler, and we will report it that way.

Why not just adapt the mass matrix from the warmup sample covariance, as standard tools do? This is the real threat. On unimodal Gaussians a dense covariance estimate converges to the right answer and may tie with our method. Our expected edge has two sources. First, the probe recovers stiff directions from a single trajectory, whereas covariance estimates need many roughly independent draws, which is slow in high dimension. Second, a global covariance blends modes with different shapes. The warmup-budget sweep and the curvature-mismatched mixture experiment below are designed to decide this. If covariance adaptation matches us at every budget, the proposal reduces to a cheaper replacement for Hessians in AD-HMC only.

Why not just add Hessian-driven damping or momentum restarts, as optimization does? Damping changes the dynamics, and in Gradient-adjusted underdamped Langevin dynamics for sampling it comes with Euler-Maruyama discretization bias and no Metropolis correction. Restarts in Practical Perspectives on Symplectic Accelerated Optimization succeed by discarding energy, which pulls a sampler toward the mode rather than across the distribution. We use the oscillations and leave the exact, energy-conserving dynamics untouched. We include the damped Langevin baseline so that this choice is tested, not assumed.

Why not just switch to logarithmic kinetic energy, which slows particles in high-density regions? Hamiltonian Dynamics with Non-Newtonian Momentum for Rapid Sampling reports that this approach still struggles on the 20-dimensional funnel, where scale changes across dimensions. That is a metric problem, which kinetic-energy shape does not address. We include the funnel to see whether our residual check correctly declines to adapt where the quadratic model fails.

Proposed Experiments

The first experiment checks estimator accuracy. On a 250-dimensional correlated Gaussian with controlled condition numbers, we measure relative error of the recovered top-$r$ eigenvalues and principal angles against the true Hessian, as a function of buffer length and $\epsilon^2\lambda_{\max}$. The prediction from the recurrence identity is near-exact recovery of the stiff directions and systematic misses of directions whose rotation angle is below roughly $2\pi/k$.

The second experiment is sampling efficiency. We run all baselines on the 250-dimensional Gaussian, the S&P 500 stochastic volatility model, German credit logistic regression, and Gaussian mixtures in 20 to 100 dimensions whose components have deliberately mismatched covariances. We report minimum and median effective sample size per gradient (warmup included), energy-distance or MMD to exact reference samples, and warmup cost in gradient equivalents. The decisive plot is effective sample size versus warmup budget on the mismatched mixture. Our method should reach its plateau earliest, and sample-covariance NUTS should plateau lower.

The third experiment covers downstream Bayesian neural network regression on UCI Concrete and CCPP, reporting RMSE and calibration against the gradient-adjusted Langevin figure above.

The fourth experiment tests failure regimes: the 20-dimensional funnel and the piecewise targets used in Non-Volume Preserving Hamiltonian Monte Carlo and No-U-TurnSamplers. Here we measure how often the residual and forward-backward agreement checks reject estimates, and whether rejecting them keeps performance at the diagonal-fallback level rather than below it.

The proposal is falsified in its interesting form if, at matched total gradients, sample-covariance adaptation ties it on the mismatched mixtures. It is also falsified if the random-direction ablation performs as well, which would mean the oscillations carry no information beyond generic low-rank structure.