Which Way Momentum Breaks: The Type, Not the Location, of Its Stability Edge

When a momentum optimizer's step size is pushed past the point where the local curvature can be tolerated, one of two things happens. Either the iterates settle into a steady back-and-forth oscillation across a valley, or they are thrown out of the basin entirely, landing somewhere flatter after a loss spike (a "catapult") or diverging. Linear stability analysis, which studies the update rule linearized around a minimum, tells us exactly where this boundary lies. For heavy-ball momentum it lies at sharpness $2(1+\beta)/\eta$. It says nothing about which of the two outcomes follows. We propose to answer that question. The tool is the next-order expansion of the update map at the boundary, known as the bifurcation normal form. It yields one measurable number whose sign separates bounded oscillation from escape.

The idea has a simple backbone that, as far as the file shows, has not been written down. The period-2 oscillations of heavy-ball momentum are exactly the period-2 oscillations of plain gradient descent run at step size $\eta/(1+\beta)$, in any dimension. It follows that near the boundary, heavy ball should behave like rescaled gradient descent. Genuinely momentum-specific behavior must then come from three sources. The first is Nesterov's look-ahead gradient. The second is nonlinear velocity dampers. The third is dynamics far from the boundary. This gives a sharp, falsifiable account of the three research directions. Linear stability sets where the edge is. Momentum-induced oscillation corresponds to a supercritical crossing, where a small stable oscillation appears. Momentum-induced divergence and catapults correspond to a subcritical crossing, where no small stable oscillation exists and the iterates escape.

Core Claim

Write heavy-ball momentum as $x_{t+1} = x_t - \eta \nabla L(x_t) + \beta (x_t - x_{t-1})$, where $L$ is the loss, $\eta$ the step size and $\beta$ the momentum coefficient. Suppose the iterates alternate between two points $a$ and $b$. Substituting into the update twice gives

$$(1+\beta)(b-a) = -\eta \nabla L(a), \qquad (1+\beta)(a-b) = -\eta \nabla L(b).$$

These are exactly the conditions for a period-2 orbit of gradient descent with step $\eta/(1+\beta)$. The heavy-ball fixed point loses stability through an eigenvalue crossing $-1$ at sharpness $2(1+\beta)/\eta$. For a one-dimensional quadratic, the product of the two roots of the characteristic polynomial equals $\beta<1$, so complex roots cannot reach the unit circle. The crossing is therefore always a period-doubling (flip) bifurcation, never the quasi-periodic Neimark–Sacker type raised in the file's Gap 2.

Because the bifurcating branch of orbits is identical to that of rescaled gradient descent, we expect the criticality to match as well. Criticality is whether the stable oscillation appears on the unstable side of the boundary (supercritical) or no small stable oscillation exists there (subcritical). We will prove this match via center-manifold reduction, stating the non-degeneracy conditions under which it holds. The claim is testable. Just past the boundary, heavy ball at $(\eta,\beta)$ and gradient descent at $\eta/(1+\beta)$ should show the same oscillation amplitude, which should grow like the square root of the distance past the threshold. They should also show the same super/subcritical sign. That sign is determined by the third and fourth directional derivatives of $L$ along the top Hessian eigenvector, plus a correction for coupling to other directions.

Three modifications break the equivalence in computable ways. For Nesterov momentum, the gradient on a period-2 orbit is evaluated at the look-ahead point $a+\beta(a-b)$, not at $a$. This shifts the orbit equations and hence the normal-form coefficient.

For the nonlinear dampers, the key fact is that their linearization at zero velocity is zero. Cubic damping of the momentum (Adaptive Momentum and Nonlinear Damping) and the velocity-dependent brake of VRAdam (Velocity Regularized Adam) are therefore invisible to linear stability analysis. They cannot move the sharpness plateau. They act only on the size and type of the oscillation. For the VRAdam-style brake $\eta_t = \alpha_0/(1+\beta_3\|v_t\|^2)$, where $v_t$ is the velocity, a period-2 orbit has constant speed $\|a-b\|$. Its orbit equations are those of gradient descent with an amplitude-dependent step:

$$\eta_{\text{eff}}(a,b) = \frac{\alpha_0}{(1+\beta)\,(1+\beta_3 \|a-b\|^2)}.$$

This adds a term of known sign to the cubic coefficient of the normal form. The damper pushes the crossing toward supercritical and shrinks the oscillation.

Relation to Prior Work

Four papers motivate the proposal by locating the boundary without typing it. Momentum Further Constrains Sharpness at the Edge of Stochastic Stability derives mean-square linear thresholds for heavy-ball and Nesterov momentum. It reports plateaus at $2(1+\beta)/\eta$ for large batches and $2(1-\beta)/\eta$ for small batches. Its analysis is linear, so it cannot say whether crossing produces ringing or a catapult. It also cannot see cubic dampers at all. When and Why Momentum Accelerates SGD finds empirically that momentum tolerates $(1+\mu)/(1-\mu)$ higher sharpness through an effective learning rate. Our orbit identity is the exact local statement behind a version of that rescaling. We then ask where the rescaling stops being exact. Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults shows that momentum produces larger catapults and overshoots at $\beta=0.99$, proven only for a scalar ReLU model. We supply a criterion, the normal-form sign, that predicts when a crossing becomes a catapult. We also directly test whether momentum's larger catapults survive a matched-effective-step comparison.

One paper supplies the method. Trajectory Alignment models gradient descent at the edge of stability as a period-doubling bifurcation, with proofs for two-layer linear networks on one data point. We reuse its bifurcation viewpoint and its toy setting. Our extension is to the $2d\times 2d$ momentum system and to an estimator usable on real networks.

Three damping methods constrain what a useful outcome looks like: cubic damping and iKFAD, VRAdam, and Aggregated Momentum, which averages several velocity buffers. All three report gains, but their damping constants are set by trial and error. The file names this as an open problem, noting sensitivity to the constants $c$ and $\rho$ in cubic damping. Our normal-form coefficient gives a principled target for those constants.

Finally, Algorithmic Instabilities of Accelerated Gradient Descent proves exponential path divergence for Nesterov momentum on smooth convex functions. Its worst-case construction uses piecewise-constant Hessians, for which third derivatives, and hence the normal form, are undefined. Our analysis assumes a loss with three continuous derivatives. That assumption is one candidate explanation for why practical networks avoid the worst case. We treat it as a hypothesis, not a result.

Research Program

The theoretical part has three pieces. The first is the heavy-ball orbit identity together with a center-manifold proof that criticality transfers from gradient descent at step $\eta/(1+\beta)$. This proof will state the non-degeneracy conditions explicitly. The second is the corrected normal-form coefficient for Nesterov momentum. The third is the coefficient contributions of cubic damping, the VRAdam brake and aggregated momentum. Aggregated momentum is linear, so it does move the threshold. Its buffer-agreement mechanism should appear as a changed linear spectrum, which cleanly separates it from the two nonlinear dampers.

The computational part is an estimator of the normal-form coefficient for a trained network at a given point. The procedure is as follows. Find the top Hessian eigenvector $u$ with power iteration on Hessian-vector products. Estimate the third and fourth derivatives of $L$ along $u$ by finite differences of Hessian-vector products. Add the leading correction from coupling to other directions, which requires one conjugate-gradient solve with the Hessian restricted orthogonally to $u$. The cost is a few dozen Hessian-vector products, far below the full distributional Hessian diagnostics that Andreyev et al. note do not scale.

The algorithmic part turns the coefficient into a rule for setting the cubic-damping constant, or VRAdam's $\beta_3$. Choose the smallest constant that makes the crossing supercritical with a target oscillation amplitude, re-estimating occasionally during training.

Baselines

The naive reference point is linear stability alone. It predicts that crossing $2(1+\beta)/\eta$ always yields bounded oscillation, or equivalently it uses the majority outcome observed in a sweep. It is implemented from the same sharpness measurements our method uses. Beating it means classifying oscillation versus catapult/divergence outcomes substantially better than the majority rate on held-out step-size sweeps.

The strongest predictive competitor is the effective-learning-rate view of Fu et al.. Under this view, momentum runs are predicted from gradient-descent runs at step $\eta/(1-\mu)$, or at $\eta/(1+\beta)$ as our identity suggests. It is the strongest competitor because our own theory says it is exact for heavy ball near the boundary. We will run it with identical networks, data and seeds, with gradient descent at the rescaled step as a separate arm. Our proposal beats it only where the theory says it should: for Nesterov, for the dampers, and for how oscillation amplitude scales with the excess past threshold. If rescaled gradient descent also matches those, our momentum-specific corrections are unnecessary.

For damping selection, the strongest existing methods are grid-tuned cubic damping, VRAdam and aggregated momentum. Adaptive Momentum and Nonlinear Damping reports DistilBERT on SST-2 test loss of 0.268 for cubic damping versus 0.299 for Adam. Velocity Regularized Adam reports GPT-2 (124M) FineWebEdu-10B validation loss of 3.447 versus 3.511 for AdamW. Aggregated Momentum reports MNIST autoencoder training MSE of 1.39 versus 2.51 for classical momentum. Each grid-tuned method gets the same total tuning compute as our method's estimation overhead plus a small final sweep. Beating these means matching their final metric within seed noise while using a fraction of the tuning runs, or surpassing them at equal tuning budget.

Two ablations isolate the key ingredient. In the first, the damping constant is chosen from the linear spectrum only, which by construction gives no information. In the second, the coefficient is estimated along a random direction instead of the top eigenvector.

Why not just…?

Why not just use the effective learning rate and gradient-descent edge-of-stability theory? This is the real threat, and our own identity partly endorses it. For heavy ball near the boundary, rescaled gradient descent should be exactly right. The proposal's claim is narrower and checkable: the rescaling fails in computable ways for Nesterov momentum and for nonlinear dampers, and possibly far past the threshold. The matched-effective-step arm settles this. If catapult sizes under heavy ball match gradient descent at $\eta/(1+\beta)$, then the "momentum yields larger catapults" finding of Phunyaphibarn et al. is a rescaling effect. That would be a useful clarification in its own right.

Why not just use the existing mean-square linear analysis? Andreyev et al. give the thresholds, and we adopt them for the location of the boundary. A linearized operator cannot distinguish outcomes that differ only at cubic order, and it assigns zero effect to cubic damping. Our prediction that cubic damping leaves the sharpness plateau unchanged while shrinking the oscillation is exactly what their analysis cannot produce. That prediction also falsifies our account if the plateau moves.

Why not just grid-search the damping constants? Grid search works; the reported gains of cubic damping and VRAdam come from it. Its cost grows with every new task, which is the brittleness Karoni et al. report. If our estimator does not cut the number of tuning runs at equal final quality, the algorithmic contribution fails, and the proposal reduces to an analysis paper.

Why not just lower $\beta$ or add warm-up? Lowering $\beta$ moves the boundary down by the factor $(1+\beta)$. It does not change which outcome follows a crossing, and it forfeits the tolerance for sharpness documented by Fu et al.. Warm-up changes how quickly the boundary is approached, not what happens on crossing it. We include $\beta\in\{0, 0.9\}$ arms so the reader can see both effects directly.

Proposed Experiments

We start where proofs are possible. The first setting is the single-point two-layer linear network of Trajectory Alignment, and the second is the scalar ReLU model of Phunyaphibarn et al.. In both we verify the orbit identity numerically, the square-root growth of amplitude with excess step, and the predicted sign changes under Nesterov momentum and the dampers.

We then move to full-batch training on a fixed subset of CIFAR-10 with MLPs, CNNs and a small ResNet. At many checkpoints we estimate the coefficient, then run short continuations at step sizes slightly above the local threshold. The decisive quantities are three. The first is the accuracy with which the coefficient's sign predicts bounded oscillation versus catapult or divergence, measured as AUROC against the linear-only and effective-step baselines. The second is agreement between predicted and measured amplitude slopes. The third is whether heavy ball and gradient descent at $\eta/(1+\beta)$ coincide near threshold while Nesterov does not.

The damping test runs cubic damping and a VRAdam-style brake on the same CIFAR-10 setup at several constants. We expect the sharpness plateau to stay at $2(1+\beta)/\eta$ as the constant varies, while oscillation amplitude falls as predicted. Aggregated momentum, being linear, should move the plateau.

We will sweep batch size to find where the deterministic theory breaks down. It should give way to the small-batch plateau of $2(1-\beta)/\eta$ reported by Andreyev et al., which lies outside our claims.

Finally, the damping-selection rule is evaluated on DistilBERT on SST-2 and on GPT-2 (124M) on FineWebEdu-10B, against the grid-tuned numbers above at matched tuning budgets. For CIFAR-10 test accuracy of the damping methods, no reported result was found on Lacuna, so we will establish those baselines ourselves.

The proposal fails cleanly in four cases. The first is if the coefficient's sign predicts outcomes no better than the linear threshold. The second is if the dampers move the plateau. The third is if rescaled gradient descent matches Nesterov and the damped variants as well as it matches heavy ball. The fourth is if coefficient-guided damping needs as many tuning runs as grid search.