The Instability and Fragility of Momentum Acceleration

15 concepts7 papers1 proposal

Momentum-based optimization methods, such as Nesterov’s Accelerated Gradient (NAG) and the Heavy-Ball method, are considered the gold standard for training machine learning models. By accumulating velocity across iterations, these algorithms smooth out local oscillations and accelerate through flat valleys, achieving theoretically optimal convergence rates in idealized settings.

However, this acceleration comes with a hidden cost: fragility. In realistic optimization landscapes—characterized by varying curvature, stochastic mini-batch noise, and nonsmooth boundaries—the very mechanism that drives acceleration can cause the optimizer to overshoot, oscillate violently, or diverge exponentially to infinity. Understanding why momentum fails in these settings requires looking closely at how accumulated velocity interacts with changing geometric landscapes.

The Slingshot Effect: Curvature Variation and Exponential Divergence

The core intuition behind momentum is physical: a particle with mass continues moving in its current direction, resisting sudden changes. While this inertia helps traverse flat regions, it becomes dangerous when the curvature of the objective function changes abruptly.

Imagine two optimization trajectories starting with slightly different initial velocities. In a perfectly quadratic landscape, this small difference remains bounded. However, as proven by Attia et al., when navigating regions of varying curvature, NAG's stability can deteriorate exponentially with the number of iterations.

This happens due to a geometric "slingshot effect." Because the two trajectories have slightly different velocities, they encounter high-curvature boundaries at slightly different times. The trajectory that hits the high-curvature wall first receives a massive, sudden corrective gradient while the other trajectory is still accelerating through the low-curvature zone. This asynchronous interaction amplifies their initial state differences exponentially, pushing the paths apart and destabilizing the optimization process.

From a physical perspective, França et al. show that classical momentum methods are discretizations of conformal Hamiltonian systems. In steep landscapes or functions with fast-growing tails, the continuous accumulation of momentum translates into excessive kinetic energy. Without a physical "speed limit," the optimizer accumulates too much velocity to turn or stop, causing it to fly off the landscape entirely.

The Mini-Batch Trap: Noise and Catastrophic Overshooting

The fragility of momentum becomes even more pronounced when moving from deterministic optimization to stochastic mini-batch training. In a full-batch setting, the gradient represents the true, smooth average landscape. In a mini-batch setting, each step is guided by a noisy approximation of that landscape.

As Assran et al. demonstrate, an accelerated algorithm tuned for deterministic optimality can diverge to infinity in finite-sum (mini-batch) settings. The mechanism behind this failure is a form of directional overshooting:

  1. Velocity Accumulation: The optimizer moves through a sequence of mini-batches that happen to share a low-curvature direction. Momentum builds up a massive velocity along this path.
  2. The Curvature Mismatch: The algorithm suddenly encounters a mini-batch with high local curvature (a sharp "wall" or "valley" in the data distribution).
  3. Catastrophic Overshooting: Because of its accumulated inertia, the optimizer cannot adapt instantly to the sudden high gradient. It overshoots the local minimum catastrophically, leading to an explosion in the loss value.

To prevent this divergence, practitioners are often forced to tune hyperparameters so conservatively that the theoretical speedup of acceleration vanishes entirely.

The Critical Thresholds for Acceleration

Why does momentum succeed in some deep learning workflows but fail in others? Recent research shows that momentum does not provide a smooth, continuous benefit; instead, its behavior is governed by sharp phase transitions and thresholds.

The Batch Size Threshold

To prevent noise from causing momentum to over-correct and diverge, the mini-batch size must be large enough to accurately resolve the landscape's geometry. Dang et al. prove that stochastic momentum methods require a minimum mini-batch size that is directly proportional to the problem's condition number (the ratio of maximum to minimum curvature). If the batch size falls below this critical threshold, the gradient noise is amplified by the momentum buffer, causing exponential divergence rather than acceleration.

Similarly, Cheng et al. identify a sharp phase transition dictated by what they term the Implicit Conditioning Ratio (ICR). Below this batch size threshold, the noise dominates, and momentum performs identically to standard Stochastic Gradient Descent (SGD). Only when the batch size exceeds this threshold does the algorithm transition into the accelerated regime typical of full-batch optimization.

The Learning Rate Threshold

A parallel threshold exists for the learning rate. Wang et al. demonstrate empirically that momentum only provides a training speedup over standard SGD when the effective learning rate exceeds a specific high-update threshold. Below this value, the updates are too small for inertia to meaningfully alter the trajectory, rendering the momentum buffer redundant.

Mini-Batch Size / Learning Rate
  │
  ▼ High (Above Threshold)  ──► Smooths noise, enables true acceleration
  │
  ▼ Low  (Below Threshold)   ──► Noise dominates ──► Overshooting & Exponential Divergence

Hidden Divergence in Automatic Differentiation

The instability of momentum is not just a problem for finding a model's parameters; it also fundamentally disrupts our ability to analyze those parameters using differentiable programming.

When training modern architectures, developers often use automatic differentiation (AD) to compute derivatives through an iterative optimization process (e.g., for hyperparameter optimization or meta-learning). Bolte et al. uncover a striking paradox: even when a momentum-based algorithm successfully manages to navigate a nonsmooth landscape and converge to the correct solution, the associated derivatives computed via AD can still oscillate violently or explode.

This occurs because the backpropagation path through an accelerated algorithm tracks the history of the momentum buffer. The rapid directional changes and inertial corrections that allow the optimizer to find a minimum create highly sensitive, chaotic intermediate states. As a result, while the primary trajectory stabilizes at a minimum, the Jacobian matrix tracking the sensitivity of that trajectory diverges, making gradient-based meta-learning fundamentally unstable when paired with aggressive acceleration.

Key Foundational Work

Algorithmic Instabilities of Accelerated Gradient Descent (2021) — contributed 3 concept elements to this direction.
On the Convergence of Nesterov's Accelerated Gradient Method in Stochastic Settings (2020) — contributed 2 concept elements to this direction.
Conformal Symplectic and Relativistic Optimization (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_momentum_induced_divergence_c5186add,
  title = {momentum-induced divergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-induced-divergence/rdr_c5186add422f48ad8574012a3074e18c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_momentum_induced_divergence_c5186add_v1,
  title = {momentum-induced divergence},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/momentum-induced-divergence/rdr_c5186add422f48ad8574012a3074e18c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake