In deep learning, momentum is almost universally treated as a force for acceleration. By adding a fraction of the previous step’s velocity to the current gradient update, classical techniques like Polyak’s Heavy Ball or Nesterov Accelerated Gradient help optimizers speed through flat plateaus and damp high-frequency oscillations. This intuition works beautifully when the goal is simple function minimization.
But what happens when optimization becomes a competitive game? In frameworks like Generative Adversarial Networks (GANs), actor-critic reinforcement learning, or constrained optimization via Lagrange multipliers, we are no longer minimizing a single objective. Instead, we are searching for a saddle point or a Nash equilibrium in a multi-agent min-max game. In these landscapes, traditional positive momentum famously breaks down—often acting as negative friction that destabilizes the system and causes parameters to spiral out of control.
To fix this, a growing body of research turns classical intuition on its head by introducing negative momentum. Rather than accelerating the optimizer's velocity, negative momentum acts as a braking mechanism or physical friction, absorbing excess kinetic energy to stabilize otherwise untrainable dynamics.
The Geometry of Games: Why Positive Momentum Fails
To understand why negative momentum works, we must first look at why positive momentum fails in min-max settings. In a standard minimization problem, the gradient field points directly toward the local minimum. In a zero-sum game, however, the interaction between competing objectives introduces strong rotational forces. Mathematically, the Jacobian of the gradient field contains large imaginary eigenvalues, meaning the vector field circles around the equilibrium point rather than pointing directly at it.
Imagine a particle caught in a whirlpool. If you throw a heavy ball with high positive momentum into this rotational field, the built-up velocity flings the particle outward. As Gauthier Gidel et al. demonstrate in Negative Momentum for Improved Game Dynamics, applying standard positive momentum to these rotational fields causes the optimizer to overshoot the equilibrium and spiral outward toward divergence.
Instead of dampening oscillations, positive momentum acts as "negative friction," a phenomenon analyzed by Motahareh Sohrabi et al. in On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization. It pumps energy into the system, forcing the iterates into divergent limit cycles rather than settling into a stable saddle point (Guojun Zhang et al. in Convergence of Gradient Methods on Bilinear Zero-Sum Games). This explains why standard deep learning optimizers like Adam often fail or require highly sensitive hyperparameter tuning when applied to adversarial games.
Negative Momentum as a Damping Force
To counteract these rotational forces, we must introduce actual physical friction. This is exactly what negative momentum does. In its simplest implementation, a negative momentum parameter subtracts a fraction of the previous update from the current step, actively resisting the velocity built up by the system.
As Zijian Fang explains in Rapid Learning in Constrained Minimax Games with Negative Momentum, this subtraction acts as a dampening mechanism that absorbs the kinetic energy driving the cyclic instability. Instead of orbiting the saddle point indefinitely, the negative momentum forces the trajectory to spiral inward toward the stable Nash equilibrium.
This continuous-time behavior is thoroughly mapped out by Yi Feng et al. in Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games, who show that lowering or reversing momentum parameters expands the permissible step-size range and prevents the optimizer from being flung out of saddle points.
This core insight has manifested in several algorithmic variations across the literature:
- Momentum Reversal in Adaptive Optimizers: To stabilize Adam in zero-sum games, Yi Feng et al. show in Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach that one must reverse standard parameter conventions by using a very low or negative first-order momentum combined with a high second-order momentum.
- Complex Momentum: Jonathan Lorraine et al. generalize momentum to be complex-valued in Complex Momentum for Optimization in Games, allowing the optimizer to generate controlled counter-rotations that precisely cancel out the frequencies of the game's rotational dynamics.
- Implicit Damping via Lookahead and Conjugate Gradients: Other methods achieve a similar damping effect implicitly. For instance, Hiroki Naganuma et al. utilize historical search directions in Conjugate Gradient Method for Generative Adversarial Networks to prevent overshooting saddle points, while Tatjana Chavdarova et al. use linear interpolation between weight states in Taming GANs with Lookahead-Minmax to suppress the vector field's rotational components. Similarly, Constantinos Daskalakis et al. demonstrate in The Limit Points of (Optimistic) Gradient Descent in Min-Max Optimization that utilizing the previous iteration's gradient acts as a negative momentum term that expands the region of attraction for stable limit points.
Anchoring Iterates: Negative Momentum for Variance Reduction
While negative momentum is a natural fit for adversarial games, a parallel line of research leverages it to solve a completely different problem: stochastic noise in finite-sum empirical risk minimization.
In large-scale machine learning, stochastic gradient methods suffer from high variance because they only sample a tiny fraction of data points or features at each step. To accelerate these methods without letting stochastic noise derail the optimization path, algorithms like Katyusha or SAGA use a form of negative momentum.
Here, negative momentum does not damp game-theoretic rotations; instead, it acts as an anchor. As Zebang Shen et al. show in Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization and Pooria Joulani et al. describe in A simpler approach to accelerated optimization: iterative averaging meets optimism, the negative momentum term shifts or pulls the current query point back toward a stable reference snapshot or moving average of historical gradients.
Imagine trying to walk along a windy ridge. Nesterov acceleration speeds you up, but a sudden gust of wind (a high-variance gradient sample) can fling you off the cliff. Negative momentum acts like a bungee cord tied to a secure anchor point behind you. If a stochastic step tries to fling you too far into a high-variance zone, the anchor pulls you back, stabilizing the accelerated path (Aymeric Dieuleveut et al. in Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression).
For example, Kaiwen Zhou et al. use this in Direct Acceleration of SAGA using Sampled Negative Momentum to pull iterates toward a randomly selected entry in a gradient history table, achieving direct acceleration without needing periodic full-gradient computations. In non-deterministic settings like entropic Optimal Transport, Yiling Luo integrates a similar stabilization term in Improved Rate of First Order Algorithms for Entropic Optimal Transport to pull estimates toward a moving average, preventing overshooting due to stochastic noise.
Theoretical Limits and Broader Frameworks
Is negative momentum a silver bullet for all saddle-point problems? Not quite. While it provides a simple and effective practical tool for taming oscillations, the authors of On the Suboptimality of Negative Momentum for Minimax Optimization prove that even with optimal tuning, negative momentum only achieves an iteration complexity of $O(\kappa^{1.5})$, which falls short of the optimal $O(\kappa)$ rate delivered by more complex methods like Extra-gradient (EG).
Nonetheless, its ease of implementation makes it a cornerstone of modern variational inequality solvers (Aleksandr Beznosikov et al. in Smooth monotone stochastic variational inequalities and saddle point problems: A survey; Alexander Pichugin et al. in Method with batching for stochastic finite-sum variational inequalities in non-Euclidean setting) and distributed optimization frameworks (Aleksandr Beznosikov et al. in Compression and Data Similarity: Combination of Two Techniques for Communication-Efficient Solving of Distributed Variational Inequalities), offering a robust bridge between the physics of damping and the mathematics of competitive optimization.
Key Foundational Work
Accelerated Doubly Stochastic Gradient Algorithm for Large-scale Empirical Risk Minimization (2017) — contributed 1 concept elements to this direction.
Katyusha: the first direct acceleration of stochastic gradient methods (2017) — contributed 1 concept elements to this direction.
Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regression (2016) — contributed 1 concept elements to this direction.