Many quantities that machine learning cares about live on curved spaces rather than flat vector spaces. Examples include covariance matrices, which form the manifold of symmetric positive definite (SPD) matrices, and tree-like embeddings, which live in hyperbolic space. Both spaces have non-positive curvature: straight lines (geodesics) spread apart instead of converging, and any two points are joined by exactly one shortest path. When we want uncertainty over such objects, the standard tool is stochastic-gradient Langevin dynamics, a sampler that takes noisy gradient steps plus injected noise and, with small enough steps, wanders according to the posterior distribution. Its weak point is the step size. A step small enough to survive sharp, ill-conditioned regions, such as nearly singular covariance matrices or points far out toward the boundary of hyperbolic space, wastes most of the computation everywhere else. A larger step diverges.
We propose to let the sampler change its own clock. The idea is to slow time down where the gradient is large and speed it up where the landscape is flat, while adding one cheap correction term so that the target distribution is exactly preserved. This idea, physical time-rescaling, already works in flat parameter spaces. We argue it transfers to non-positively curved manifolds almost for free, and that non-positive curvature is what makes the transferred method provably stable without confining it to a bounded region. The deliverable is a sampler with a proof of long-run stability and a measured advantage, at equal gradient budget, over the best-tuned fixed-step Riemannian sampler on SPD and hyperbolic posteriors.
Core Claim
The Euclidean starting point is Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks. That paper's SA-SGLD sampler rescales the time variable of Langevin diffusion using a monitor of the stochastic gradient norm. It navigates high-curvature funnels where plain SGLD diverges, and it keeps the correct posterior, unlike preconditioned heuristics such as pSGLD. It is restricted to $\mathbb{R}^d$. In parallel, On Riemannian Stochastic Approximation Schemes with Fixed Step-Size (see also the companion entry) shows the following for fixed-step Riemannian stochastic gradient schemes on non-compact non-positively curved manifolds, which we call Hadamard manifolds. The squared distance to the solution can be "Huberized" into a Lyapunov function, a quantity guaranteed to decrease on average. This gives geometric ergodicity with no artificial compact constraint. That paper names adaptive Riemannian schemes as open, and its stationary spread scales with $\sqrt{\eta}$ for step size $\eta$.
Our claim joins these two results. The time-rescaling correction involves only a scalar function, so it stays intrinsic on any Riemannian manifold. Write $U$ for the negative log-posterior with respect to Riemannian volume and $g(x)\in[g_{\min},g_{\max}]$ for a positive scalar monitor. The diffusion
$$dx = \big[-g(x)\,\mathrm{grad}\,U(x) + \mathrm{grad}\,g(x)\big]\,d\tau + \sqrt{2g(x)}\,dB^{\mathcal M}_\tau$$
has generator $f \mapsto e^{U}\,\mathrm{div}\!\left(g\,e^{-U}\mathrm{grad} f\right)$. It is therefore reversible with respect to $e^{-U}$ for any choice of $g$. Here $\mathrm{grad}$ and $\mathrm{div}$ are the Riemannian gradient and divergence, $B^{\mathcal M}$ is Brownian motion on the manifold, and $\tau$ is the rescaled time.
The contrast with matrix preconditioning is the whole point. A position-dependent preconditioner, the Riemannian analogue of pSGLD, needs a divergence-of-a-matrix correction that is expensive and usually dropped, and dropping it causes bias. A scalar clock needs only $\mathrm{grad}\,g$. Discretizing this diffusion with exponential-map steps of length $h\,g(x_k)$ yields a sampler whose step adapts to the state of the chain, not to a preset time schedule.
Our second claim concerns stability. Non-positive curvature gives a law-of-cosines inequality that holds for steps of any length: $d^2(\mathrm{Exp}_x v, y) \ge d^2(x,y) - 2\langle v, \mathrm{Log}_x y\rangle + \|v\|^2$. If curvature is also bounded below, a matching upper bound holds with a curvature-dependent constant. As a result, the Huberized-distance Lyapunov argument of the fixed-step analysis should go through when the step varies within $[h g_{\min}, h g_{\max}]$. We conjecture this yields ergodicity on non-compact SPD and hyperbolic spaces and an $O(h)$ bias bound, and proving it is part of the work.
Research Program
The method has three parts. The first is the monitor. We will compare two forms of $g$, both computable from quantities the sampler already has. One is a smoothed Riemannian gradient norm, as in SA-SGLD. The other adds a term in the Huberized distance to a running anchor point, which shrinks steps as the chain moves into regions where the metric is badly distorted. We will implement $\mathrm{grad}\,g$ in two ways. One uses automatic differentiation. The other lets the monitor evolve as an auxiliary scalar in the spirit of SA-SGLD, so that no second derivative is ever taken.
The second part is the integrator: an Exp-based geodesic random walk on SPD matrices with the affine-invariant metric and on the Lorentz model of hyperbolic space. A retraction variant will be included because Riemannian Adaptive Optimization Methods found retractions competitive with exact exponential maps on Poincaré embeddings.
The third part is theory. We will prove ergodicity with the variable-step Lyapunov argument and bound the weak error. We will also state precisely where curvature constants enter, and check whether they enter additively or multiplicatively. That question is open even for optimization, where the acceleration method of Accelerated Riemannian Optimization: Handling Constraints to Bound Geometric Penalties pays a multiplicative factor $\zeta^2$.
We deliberately do not pursue curvature-adaptive accelerated variational integrators. The extended-Lagrangian integrators of Time-adaptive Lagrangian Variational Integrators for Accelerated Optimization on Manifolds and Variational Symplectic Accelerated Optimization on Lie Groups adapt steps via an explicit time-dependent monitor $g(t)$ on compact groups, and the state-dependent case is an acknowledged invertibility obstacle. Separately, Negative curvature obstructs acceleration for strongly geodesically convex optimization, even with exact first-order oracles proves that plain Riemannian gradient descent is already query-optimal when curvature is bounded away from zero. Together these suggest that acceleration is the wrong target in negative curvature, and that adaptive time-stepping pays off where it controls stability and bias, not speed. Our sampler does adapt to the state of the chain, but it sidesteps the invertibility problem: it uses first-order Langevin steps, not an implicit variational map.
Baselines
The naive reference point is fixed-step Riemannian SGLD: the same Exp-based Langevin step with constant $h$, with $h$ chosen by grid search under the same gradient budget used for all methods. It is the natural "do nothing" choice, and the fixed-step analysis of On Riemannian Stochastic Approximation Schemes with Fixed Step-Size characterizes its behavior (stationary spread scaling with $\sqrt{\eta}$), though no numerical error values are reported there. We will also run it with a decreasing step schedule, the standard fix for step-size bias.
The strongest competitor is Euclidean SA-SGLD from Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks, run in a global chart. On a Hadamard manifold the logarithm map at a base point is a global diffeomorphism, so this baseline is legitimate. It uses the matrix-logarithm (log-Euclidean) chart for SPD, the tangent space at the origin for hyperbolic space, and includes the exact volume Jacobian. That paper reports no exact numbers, so no reported result was found on Lacuna for SA-SGLD in a manifold chart. We must therefore establish its performance ourselves.
The third competitor is a Riemannian preconditioned sampler in the style of pSGLD and RAdam (Riemannian Adaptive Optimization Methods). It applies a per-factor adaptive scale with the matrix-divergence correction dropped, as practitioners do.
All methods get identical data, minibatches, gradient-evaluation budgets, and equal tuning grids, with three to five values per hyperparameter. Exact-gradient Riemannian Langevin with Metropolis correction, run long, serves as the ground-truth sampler on targets without closed-form posteriors.
Our ablation removes the $\mathrm{grad}\,g$ correction while keeping the adaptive step. This isolates the key ingredient. If it shows bias that does not shrink with $h$ while the corrected sampler's bias does, the correction is doing real work.
Beating the strongest baseline means the following at matched gradient budget. Posterior-mean error, measured as geodesic distance from the reference Fréchet mean, and posterior-spread error must be at least 30% lower than both the best-tuned fixed-step sampler and the chart-based SA-SGLD. The method must also show no divergences across an initial-step range spanning two orders of magnitude.
Why not just…?
Why not just tune a fixed step carefully? Tuning works when the posterior is uniformly conditioned. It fails when one target contains both flat regions and sharp regions, such as heavy-tailed priors pushing covariances toward singularity. There, the stable fixed step is set by the worst region, and the fixed-step bias scales with that step everywhere. The heavy-tailed SPD experiment is designed to make exactly this trade-off visible. If a single tuned step matches our sampler there, the proposal's practical case fails.
Why not just run Euclidean SA-SGLD in a global log chart, since Hadamard manifolds have one? This is the most serious threat. The chart is exact in principle, but the metric it pulls back distorts the geometry in a way that grows exponentially with distance from the base point in negative curvature. Gradient norms measured in the chart then misreport local stiffness, and the volume Jacobian becomes badly scaled. Evidence that chart-based modeling degrades sharply is indirect. The examples are stereographic flows in Riemannian Continuous Normalizing Flows, with an NLL of 63.60 against -1.78 on a synthetic sphere target, and tangent-space aggregation in Differentiating through the Fréchet Mean, with a Disease ROC AUC of 76.4 against 93.7. Neither involves Langevin sampling on a Hadamard manifold. We will therefore settle the question by sweeping how far the posterior mass sits from the chart's base point. Our prediction is that the intrinsic sampler's advantage grows with that distance. If the chart method matches it at all distances, we will report that the intrinsic construction is unnecessary.
Why not just use a Riemannian preconditioner like RAdam or pSGLD? Matrix preconditioning needs a divergence correction to stay unbiased, and that correction is routinely omitted. SA-SGLD's own comparison shows pSGLD failing to preserve the target. On a single non-product manifold, such as one SPD matrix, the per-coordinate adaptivity of Riemannian Adaptive Optimization Methods also has no canonical coordinates to act on. The exact-posterior experiment measures this bias directly.
Proposed Experiments
The first target has an exact answer: a Bayesian covariance posterior with a conjugate inverse-Wishart prior and Gaussian data on $\mathcal{S}^{50}_{++}$. This follows the Wishart Karcher-mean setup of On Riemannian Stochastic Approximation Schemes with Fixed Step-Size, where exact posterior draws are available. The decisive check is that bias in the Fréchet mean and in the log-coordinate covariance shrinks with $h$ for our sampler and for fixed-step SGLD. The uncorrected ablation should instead plateau.
The second target is a heavy-tailed covariance posterior. It uses a multivariate $t$ likelihood with $df=2$ and 10% outlier contamination, at the $p=50$, $n=2500$ scale of Frank-Wolfe-based Algorithms for Approximating Tyler's M-estimator, together with a shrinkage prior that creates near-singular regions. The reference is long-run Metropolis-adjusted Riemannian Langevin. We measure divergence rate across initial steps, error per gradient evaluation, and effective samples per second, including the $O(p^3)$ cost of matrix decompositions.
The third target is a Bayesian Poincaré embedding of a WordNet noun subtree, following the setup of Riemannian Adaptive Optimization Methods. Here we report the calibration of link predictions and the spread of the posterior over embedding radius, where points near the boundary make the sampling stiff.
The fourth experiment is the chart-distance sweep described above. A Riemannian Gaussian target on $\mathbb{H}^{10}$ and on $\mathcal{S}^{16}_{++}$ is shifted progressively away from the chart base point.
In the zero-temperature limit the same sampler becomes adaptive stochastic gradient descent. As a secondary check, we will test whether its steady-state error on the Karcher mean falls below the linear-in-$\eta$ floor reported for fixed steps. We will not claim faster than linear rates, since Negative curvature obstructs acceleration for strongly geodesically convex optimization, even with exact first-order oracles rules them out.
The proposal succeeds if the corrected time-rescaled sampler beats the chart-based SA-SGLD in the far-from-base-point regime and beats the best-tuned fixed step on the heavy-tailed target, at equal gradient budget. It fails, and we will say so, if the chart baseline is indistinguishable everywhere.
References
- Adaptive Stepsizing for Stochastic Gradient Langevin Dynamics in Bayesian Neural Networks
- On Riemannian Stochastic Approximation Schemes with Fixed Step-Size
- Riemannian Adaptive Optimization Methods
- Accelerated Riemannian Optimization: Handling Constraints to Bound Geometric Penalties
- Time-adaptive Lagrangian Variational Integrators for Accelerated Optimization on Manifolds
- Variational Symplectic Accelerated Optimization on Lie Groups
- Negative curvature obstructs acceleration for strongly geodesically convex optimization, even with exact first-order oracles
- Riemannian Continuous Normalizing Flows
- Differentiating through the Fréchet Mean
- Frank-Wolfe-based Algorithms for Approximating Tyler's M-estimator