Stabilizing Optimization with Negative Momentum

41 concepts24 papers4 proposals

In deep learning, momentum is almost universally treated as a force for acceleration. By adding a fraction of the previous step’s velocity to the current gradient update, classical techniques like Polyak’s Heavy Ball or Nesterov Accelerated Gradient help optimizers speed through flat plateaus and damp high-frequency oscillations. This intuition works beautifully when the goal is simple function minimization.

But what happens when optimization becomes a competitive game? In frameworks like Generative Adversarial Networks (GANs), actor-critic reinforcement learning, or constrained optimization via Lagrange multipliers, we are no longer minimizing a single objective. Instead, we are searching for a saddle point or a Nash equilibrium in a multi-agent min-max game. In these landscapes, traditional positive momentum famously breaks down—often acting as negative friction that destabilizes the system and causes parameters to spiral out of control.

To fix this, a growing body of research turns classical intuition on its head by introducing negative momentum. Rather than accelerating the optimizer's velocity, negative momentum acts as a braking mechanism or physical friction, absorbing excess kinetic energy to stabilize otherwise untrainable dynamics.


The Geometry of Games: Why Positive Momentum Fails

To understand why negative momentum works, we must first look at why positive momentum fails in min-max settings. In a standard minimization problem, the gradient field points directly toward the local minimum. In a zero-sum game, however, the interaction between competing objectives introduces strong rotational forces. Mathematically, the Jacobian of the gradient field contains large imaginary eigenvalues, meaning the vector field circles around the equilibrium point rather than pointing directly at it.

Imagine a particle caught in a whirlpool. If you throw a heavy ball with high positive momentum into this rotational field, the built-up velocity flings the particle outward. As Gauthier Gidel et al. demonstrate in Negative Momentum for Improved Game Dynamics, applying standard positive momentum to these rotational fields causes the optimizer to overshoot the equilibrium and spiral outward toward divergence.

Instead of dampening oscillations, positive momentum acts as "negative friction," a phenomenon analyzed by Motahareh Sohrabi et al. in On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization. It pumps energy into the system, forcing the iterates into divergent limit cycles rather than settling into a stable saddle point (Guojun Zhang et al. in Convergence of Gradient Methods on Bilinear Zero-Sum Games). This explains why standard deep learning optimizers like Adam often fail or require highly sensitive hyperparameter tuning when applied to adversarial games.


Negative Momentum as a Damping Force

To counteract these rotational forces, we must introduce actual physical friction. This is exactly what negative momentum does. In its simplest implementation, a negative momentum parameter subtracts a fraction of the previous update from the current step, actively resisting the velocity built up by the system.

As Zijian Fang explains in Rapid Learning in Constrained Minimax Games with Negative Momentum, this subtraction acts as a dampening mechanism that absorbs the kinetic energy driving the cyclic instability. Instead of orbiting the saddle point indefinitely, the negative momentum forces the trajectory to spiral inward toward the stable Nash equilibrium.

This continuous-time behavior is thoroughly mapped out by Yi Feng et al. in Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games, who show that lowering or reversing momentum parameters expands the permissible step-size range and prevents the optimizer from being flung out of saddle points.

This core insight has manifested in several algorithmic variations across the literature:


Anchoring Iterates: Negative Momentum for Variance Reduction

While negative momentum is a natural fit for adversarial games, a parallel line of research leverages it to solve a completely different problem: stochastic noise in finite-sum empirical risk minimization.

In large-scale machine learning, stochastic gradient methods suffer from high variance because they only sample a tiny fraction of data points or features at each step. To accelerate these methods without letting stochastic noise derail the optimization path, algorithms like Katyusha or SAGA use a form of negative momentum.

Here, negative momentum does not damp game-theoretic rotations; instead, it acts as an anchor. As Zebang Shen et al. show in Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization and Pooria Joulani et al. describe in A simpler approach to accelerated optimization: iterative averaging meets optimism, the negative momentum term shifts or pulls the current query point back toward a stable reference snapshot or moving average of historical gradients.

Imagine trying to walk along a windy ridge. Nesterov acceleration speeds you up, but a sudden gust of wind (a high-variance gradient sample) can fling you off the cliff. Negative momentum acts like a bungee cord tied to a secure anchor point behind you. If a stochastic step tries to fling you too far into a high-variance zone, the anchor pulls you back, stabilizing the accelerated path (Aymeric Dieuleveut et al. in Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression).

For example, Kaiwen Zhou et al. use this in Direct Acceleration of SAGA using Sampled Negative Momentum to pull iterates toward a randomly selected entry in a gradient history table, achieving direct acceleration without needing periodic full-gradient computations. In non-deterministic settings like entropic Optimal Transport, Yiling Luo integrates a similar stabilization term in Improved Rate of First Order Algorithms for Entropic Optimal Transport to pull estimates toward a moving average, preventing overshooting due to stochastic noise.


Theoretical Limits and Broader Frameworks

Is negative momentum a silver bullet for all saddle-point problems? Not quite. While it provides a simple and effective practical tool for taming oscillations, the authors of On the Suboptimality of Negative Momentum for Minimax Optimization prove that even with optimal tuning, negative momentum only achieves an iteration complexity of $O(\kappa^{1.5})$, which falls short of the optimal $O(\kappa)$ rate delivered by more complex methods like Extra-gradient (EG).

Nonetheless, its ease of implementation makes it a cornerstone of modern variational inequality solvers (Aleksandr Beznosikov et al. in Smooth monotone stochastic variational inequalities and saddle point problems: A survey; Alexander Pichugin et al. in Method with batching for stochastic finite-sum variational inequalities in non-Euclidean setting) and distributed optimization frameworks (Aleksandr Beznosikov et al. in Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities), offering a robust bridge between the physics of damping and the mathematics of competitive optimization.

Key Foundational Work

Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization (2017) — contributed 1 concept elements to this direction.
Katyusha: the first direct acceleration of stochastic gradient methods (2017) — contributed 1 concept elements to this direction.
Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_negative_momentum_b52faab9,
  title = {negative momentum},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/negative-momentum/rdr_b52faab9c63f4c7ab59ee5b4a45e7d7b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_negative_momentum_b52faab9_v1,
  title = {negative momentum},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/negative-momentum/rdr_b52faab9c63f4c7ab59ee5b4a45e7d7b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake