Langevin samplers and noisy gradient optimizers move a point downhill along the gradient while adding random noise. A known trick makes them faster: add a "rotational" push that circulates probability around level sets without changing the distribution the sampler eventually settles into. In practice this push is chosen blindly, as a fixed antisymmetric matrix scaled by one knob. If the knob is too small, nothing improves. If it is too large, the simulation blows up. We propose to treat the useful part of this rotation as something borrowed from physics: a tachyonic mode, meaning a direction of negative curvature (an "upside-down" mass term) along which perturbations grow exponentially. Rotational coupling makes those modes grow faster, which is exactly what escaping a saddle point requires. It also makes the stable directions stiffer and oscillatory, and that is what breaks the discretization.
The proposal is to put the rotation only where it pays. We estimate a few negative-curvature and positive-curvature directions with cheap Hessian-vector products, couple only those directions, and size the coupling using a geometric stability bound on the discretized operator rather than its eigenvalues. Stated in two sentences: non-reversible speedups in non-convex problems come mostly from amplifying the growth of unstable saddle directions, and the instability that limits them comes from rotation in the stable directions. Aiming the rotation at the unstable subspace should therefore keep the speedup while letting the step size stay large.
Core Claim
Non-Convex Optimization via Non-Reversible Stochastic Gradient Langevin Dynamics adds an antisymmetric matrix $J$ to the drift, giving $dX_t = -(I+J)\nabla f\,dt + \sqrt{2\beta^{-1}}dB_t$. This preserves the Gibbs target and enlarges the spectral gap. The paper and its constrained successor, Non-Reversible Langevin Algorithms for Constrained Sampling, both report that performance hinges on the scale $\tau$ of $J$. Both leave the choice of $J$ open, and both use $J$ that is generic rather than adapted to the landscape. Near a critical point with Hessian $H$, the linearized drift is $-(I+J)H$.
Two elementary facts organize the proposal. The first is that $\mathrm{tr}(JH)=0$ for antisymmetric $J$ and symmetric $H$. Rotation therefore cannot change the total growth rate. It can only redistribute growth among directions. The second comes from coupling one positive-curvature direction (curvature $a>0$) to one negative-curvature direction (curvature $-b<0$) with strength $t$. The escape rate along the unstable direction then rises from $b$ to
$$\lambda_{\text{esc}}(t) = \tfrac{1}{2}\left[(b-a) + \sqrt{(a+b)^2 + 4abt^2}\right],$$
which follows from the trace $a-b$ and determinant $-ab(1+t^2)$ of the $2\times 2$ block of $(I+J)H$. Escape grows roughly as $t\sqrt{ab}$.
The cost appears on the other side. Explicit Euler steps remain stable for a mode with eigenvalue $\lambda$ of $(I+J)H$ only if $|1-\eta\lambda|<1$, which for $\mathrm{Re}\,\lambda>0$ means
$$\eta < \frac{2\,\mathrm{Re}\,\lambda}{|\lambda|^2}.$$
When $J$ couples two positive-curvature directions, it adds imaginary parts but leaves real parts fixed, so the step-size ceiling collapses. This is our reading of the "too large $\tau$ explodes" behaviour. Coupling only the tachyonic subspace to a small positive-curvature partner buys escape speed at a bounded and computable stiffness cost.
Because $(I+J)H$ is non-normal, eigenvalues alone understate transient growth. This is the same failure that Wolfer and Kontorovich document for non-reversible Markov chains, where they replace the spectral gap with a pseudo-spectral gap built from a reversibilization. We therefore size $t$ using the numerical range of the coupled block, the convex region of the complex plane swept by $v^\top(I+J)Hv$ over unit vectors $v$. This is the "geometric characterization" component of the proposal. It is a characterization of the discretized operator's geometry, which in low-rank blocks is computable in closed form.
We considered and rejected a more ambitious bridge to normed-space constants such as the midline constants of Some Geometric Constants Related to the Midline of Equilateral Triangles in Banach Spaces. Nothing in the file connects them to operator stability, and forcing the link would be decoration. For the same reason we do not attempt to extend Devriendt's commute-time simplex (Kemeny’s constant and the Lemoine point of a simplex). It requires reversibility, which is exactly what we break.
Where the physics enters, and where it stops
Three lines of physics work motivate the framing. In each, tachyonic modes are produced or cured by how fields couple to curvature. Hell shows that coupling a massive vector field to the Ricci tensor generates runaway modes, and that a field-dependent redefinition of the metric removes them (Unveiling the Inconsistency of the Proca Theory with Nonminimal Coupling to Gravity). Obata finds the analogous runaway for a 2-form field (On the Kalb-Ramond field with non-minimal coupling to gravity). Goon, Hinterbichler and Trodden show that symmetry-mandated masses proportional to background curvature turn tachyonic on de Sitter backgrounds (Symmetries for Galileons and DBI scalars on curved space).
The lesson we import is narrow. Whether a negative-curvature mode runs away, and how fast, depends on what it is coupled to, and coupling is a design variable. Physics wants to suppress the runaway. A sampler wants to exploit it within a stability budget. Gap 3 of the file names this opposition.
The Ostrogradsky instability plays the role of a constraint rather than a mechanism. Ostrogradsky's theorem says that dynamics built from non-degenerate higher time derivatives have energy unbounded below. Healthy theories such as Galileons (Galileons as Wess-Zumino terms) and the ghost-free branches of Revisiting stability in new general relativity survive only because special antisymmetric structure keeps the equations second order. Our analogue concerns any modification of the sampler. It must keep a Lyapunov function bounded below, namely the free energy relative to the Gibbs target. It also must not let state-dependent, Hessian-derived drift introduce uncompensated terms.
Concretely, a $J(x)$ that varies continuously with the state requires a divergence correction to preserve the target. We avoid that by holding $J$ piecewise constant over blocks of $K$ steps, then freezing it after burn-in when the goal is sampling. Bahamonde et al. warn that linearly healthy branches can re-acquire ghosts nonlinearly. In the same spirit, we test our linearized stability rule on genuinely non-quadratic landscapes and do not trust the $2\times 2$ analysis alone.
Research Program
The algorithm, which we call targeted non-reversible Langevin (TNL), runs ordinary SGLD steps with drift $(I+J_k)g$. Every $K$ steps it refreshes $J_k$ by four operations. First, run a few Lanczos iterations with Hessian-vector products to obtain the $r$ most negative curvature directions and a matched set of $r$ positive-curvature partners, chosen to have moderate curvature rather than the largest. Second, build $J_k = \sum_i t_i(u_i w_i^\top - w_i u_i^\top)$ on those pairs. Third, choose each $t_i$ as the largest value whose $2\times 2$ block keeps the step stable, with a safety margin, using the numerical-range bound at the current $\eta$. Fourth, when no negative curvature is detected, set $J_k=0$, so the method reduces to SGLD in convex basins.
The analysis has three parts. We prove the linear escape-rate formula and its multi-pair generalization. We bound the fraction of the Euler stability region consumed by a rank-$2r$ coupling. We give a saddle-escape time bound for the discretized SDE that improves on reversible SGLD by the factor $\lambda_{\text{esc}}(t^*)/b$ at the selected $t^*$.
For sampling, we measure mixing empirically. We estimate the pseudo-spectral gap of a discretized version of the chain on low-dimensional coarse-grained state spaces, using the estimator whose minimax rate $\tilde\Theta(1/(\gamma_{\text{ps}}\pi_*))$ is established in Improved estimation of relaxation time in nonreversible Markov chains. This gives a mixing diagnostic that does not rely on the misleading complex eigenvalues.
Baselines
The naive reference is reversible SGLD at its best-tuned step size. This is the "do nothing about rotation" point. On the 9-mode Gaussian mixture it found only 3 of 9 modes in A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions.
The strongest direct competitor is NSGLD from Non-Convex Optimization via Non-Reversible Stochastic Gradient Langevin Dynamics, using its global antisymmetric $J$. We give it the same gradient-evaluation budget as TNL plus TNL's Hessian-vector-product cost converted into extra gradient evaluations. We grid-search $\tau$ and $\eta$ jointly with the same number of tuning trials as TNL's margin and $\eta$. That paper reports NSGLD beating SGLD and SGD on Iris and Diabetes Bayesian ICA and on IMDB with three architectures, but no numerical values are given; no reported result found on Lacuna for NSGLD's numeric loss or log-likelihood. Beating it means reaching NSGLD's best final loss or log-likelihood in at least 30% fewer gradient-equivalent evaluations, or tolerating a larger stable $\eta$, on a majority of benchmarks across 10 seeds.
For multimodal sampling and deep networks, the strongest existing method is CSGLD/CSGHMC from the same contour paper. It reports 9 of 9 modes on the mixture, and a CIFAR-10 accuracy of 94.06% for saCSGHMC on ResNet-20. On CIFAR-100 it reports improvements of 3.5% in best point estimate and 2% in Bayesian model averaging over momentum SGD. We run it with the authors' energy partition settings under matched epochs. Matching its mode count with fewer evaluations would count as a win. On CIFAR we expect at best parity and would report honestly if TNL falls short. CSGLD reshapes the target, while TNL leaves the target unchanged, so the two can be combined. We also test that combination.
For constrained problems, the reference is SRNLMC/SRNSGLD from Non-Reversible Langevin Algorithms for Constrained Sampling on MAGIC Gamma, Titanic, the unit-disk regression and the 3D truncated normal. That paper reports faster convergence than projected Langevin, but no numerical values; no reported result found on Lacuna for those numbers. Here TNL substitutes its targeted $J$ into the skew-reflection scheme, and the comparison is time to a fixed 1-Wasserstein error.
Two ablations isolate the key ingredient. The first is "random low-rank $J$": same rank $2r$ and same norm, but random directions. This tests whether targeting the negative curvature matters or whether low rank alone does the work. The second is "targeted $J$ with eigenvalue-based sizing instead of numerical-range sizing". This tests whether the non-normal geometric bound earns its keep.
Why not just…?
Why not just grid-search $\tau$ for a global $J$? This is the most serious threat. If a well-tuned global $J$ matches TNL, our mechanism is merely descriptive. The escape-rate and stability formulas predict that it cannot match on landscapes where the curvature spectrum is wide. There a global $J$ inevitably couples stiff positive directions, and the step-size ceiling $2\mathrm{Re}\,\lambda/|\lambda|^2$ collapses. The predicted advantage should vanish when the spectrum is narrow. We test both regimes on synthetic quadratic-plus-saddle landscapes with controlled curvature spread. If the global-$J$ baseline ties TNL across spreads, we drop the targeting claim.
Why not just use underdamped Langevin or SGHMC, which is already non-reversible? Momentum dynamics are the natural "second-order" alternative, and CSGHMC is built on them. Momentum also accelerates saddle escape, so this baseline might absorb most of the gain. We run SGHMC and SGHMC with targeted $J$ added to the position update. If targeted $J$ adds nothing on top of momentum, the honest conclusion is that TNL is a cheap first-order substitute rather than an improvement. That would still be useful where momentum tuning is fragile, but it is a narrower claim. Adding still higher-order "momentum of momentum" dynamics is precisely the pattern Ostrogradsky warns about. We do not pursue it beyond one diagnostic run.
Why not just use flat-histogram or contour reweighting for multimodality? CSGLD already gets 9 of 9 modes where SGLD gets 3. Reweighting attacks barriers between modes, whereas TNL attacks saddles along the path. These are different bottlenecks, and they predict different outcomes. We expect CSGLD to dominate on well-separated mixtures. We expect TNL to help on landscapes dominated by many shallow saddles, such as the neural network losses on IMDB. The combined CSGLD+TNL run tells us whether the two effects are complementary.
Proposed Experiments
The first decisive check is synthetic. We build $d=50$ to $1000$ landscapes with planted saddles of known curvatures $a$ and $-b$. We measure escape time against coupling strength and test whether it follows $\lambda_{\text{esc}}(t)$. We also measure the empirical divergence threshold for $\eta$ and test whether it follows the numerical-range bound better than the eigenvalue bound. Failure of either prediction falsifies the mechanism before any benchmark is run.
The second check runs on the 9-mode Gaussian mixture, Iris and Diabetes Bayesian ICA, and IMDB with the fully connected, LSTM and CNN-LSTM models. We report loss or log-likelihood against gradient-equivalent evaluations, the largest stable $\eta$, modes discovered, and effective sample size, across 10 seeds with confidence intervals.
The third check covers constrained sampling on MAGIC Gamma and Titanic, plus the unit-disk and truncated-normal targets, reporting time to fixed 1-Wasserstein error.
The fourth check is a stress test on CIFAR-10 and CIFAR-100 with ResNet-20 against the reported CSGHMC figures. It mainly tests whether Lanczos overhead stays affordable at scale.
The result that would most change our minds is the random-low-rank ablation matching TNL. That outcome would mean the benefit comes from limiting the rank of the rotation, not from aiming it at tachyonic directions.
References
- Non-Convex Optimization via Non-Reversible Stochastic Gradient Langevin Dynamics
- Non-Reversible Langevin Algorithms for Constrained Sampling
- Estimating the Mixing Time of Ergodic Markov Chains
- Some Geometric Constants Related to the Midline of Equilateral Triangles in Banach Spaces
- Kemeny’s constant and the Lemoine point of a simplex
- Unveiling the Inconsistency of the Proca Theory with Nonminimal Coupling to Gravity
- On the Kalb-Ramond field with non-minimal coupling to gravity
- Symmetries for Galileons and DBI scalars on curved space
- Galileons as Wess-Zumino terms
- Revisiting stability in new general relativity
- Improved estimation of relaxation time in nonreversible Markov chains
- A Contour Stochastic Gradient Langevin Dynamics Algorithm for Simulations of Multi-modal Distributions