In standard machine learning, we usually want to minimize a single loss function. We imagine a ball rolling down a hill, and we use momentum to help that ball gain speed, blast through small bumps, and reach the bottom faster.
However, training Generative Adversarial Networks (GANs) or solving constrained optimization problems isn't like rolling a ball down a hill. It is a game between two or more players (like a generator and a discriminator). In these "min-max" scenarios, the optimization landscape doesn't just have slopes; it has rotations. If you apply standard momentum to a rotating system, you don't reach the bottom faster—you get flung out of the orbit.
Researchers have recently discovered that the secret to stabilizing these games is often the exact opposite of what we do in standard deep learning: using negative momentum.
The Centrifugal Force of Min-Max Games
To understand why standard momentum fails, we have to look at the geometry of a game. In a simple minimization task, the gradient points toward the local minimum. In a min-max game, the players are often chasing each other in circles around a saddle point.
Abhay Kumar Yadav and Tom Goldstein describe this in Stabilizing Adversarial Nets with Prediction Methods as an undamped harmonic oscillator. Just as a frictionless pendulum swings back and forth forever without settling, standard gradient descent-ascent (GDA) tends to orbit the equilibrium (the Nash Equilibrium) indefinitely.
When we add standard (positive) momentum to this rotation, it acts like a centrifugal force. Instead of helping, the momentum carries the iterates further away from the center, causing the training to spiral out of control. Yi Feng and Volkan Cevher demonstrate in Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games that high momentum values destabilize the system by flinging the optimizer out of these saddle points.
Negative Momentum as Friction
If positive momentum is a force that pushes you outward, negative momentum is the friction that pulls you back in.
In Negative Momentum for Improved Game Dynamics, Gauthier Gidel and Simon Lacoste-Julien show that by using a negative momentum coefficient, we can introduce a damping effect. Mathematically, this shifts the eigenvalues of the system's Jacobian into a stable region.
Instead of the iterates circling the equilibrium forever, negative momentum forces the trajectory to spiral inward. Zijian Fang and colleagues apply this intuition to classic game-theory algorithms in Rapid Learning in Constrained Minimax Games with Negative Momentum, showing that subtracting a fraction of the previous update acts as a "frictional" force that settles the game much faster than standard methods.
Momentum Reversal in Adaptive Optimizers
This insight isn't just for simple gradient descent; it applies to the complex adaptive optimizers we use every day, like Adam.
In Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach, Yi Feng identifies a phenomenon called momentum reversal. To achieve stability in a zero-sum game, you actually need the opposite of the standard settings:
- Low or negative first-order momentum ($\beta_1$): To reduce the "flinging" effect.
- High second-order momentum ($\beta_2$): To provide more stable scaling.
By reversing these parameters, the optimizer stops fighting the rotational dynamics and starts converging toward the Nash Equilibrium.
Beyond GANs: Variational Inequalities
While GANs are the most famous example, this "rotational" problem exists in any system where multiple objectives interact. This is often framed as solving Variational Inequalities (VIs).
Aleksandr Beznosikov and Alexander Gasnikov have explored this extensively, noting in Smooth monotone stochastic variational inequalities and saddle point problems: A survey that negative momentum is a primary tool for suppressing oscillations in these landscapes. Similarly, in constrained optimization—where you must balance a loss function against a set of constraints—Jose Gallego-Posada and Simon Lacoste-Julien introduced $\nu$PI in On PI Controllers for Updating Lagrange Multipliers in Constrained Optimization. They argue that standard momentum acts as "negative friction" in these min-max games, and a control-theoretic approach is required to dampen the resulting oscillations.
Summary of the Intuition
| Concept | Minimization (Rolling Downhill) | Games (Chasing in Circles) |
|---|---|---|
| Dynamics | Gradient points to the goal. | Gradient points around the goal. |
| Positive Momentum | Accelerates: Helps reach the bottom faster. | Destabilizes: Flings the iterates away (Centrifugal force). |
| Negative Momentum | Slows down: Acts as an unnecessary brake. | Stabilizes: Pulls iterates toward the center (Damping/Friction). |
By moving "backward" slightly relative to the previous update, negative momentum allows the optimizer to resist the rotational pull of the game, turning a chaotic orbit into a direct path to equilibrium.