When two models are trained against each other, as in a generative adversarial network (GAN, where a generator and a discriminator compete), the updates do not simply roll downhill. Part of the update field spins around the solution. Momentum, the trick of adding a fraction of the previous step to the current one, then behaves very differently than in ordinary training. Under strong rotation, positive momentum pumps energy into the orbit and the run diverges, while negative momentum (subtracting part of the previous step) acts as friction and pulls the orbit inward. Under weak rotation, the reverse holds, and positive momentum accelerates. Every existing method fixes the momentum coefficient once, by hand, using theory that holds only near the solution. Far from the solution, the balance between spinning and descending changes from step to step.
We propose to estimate that balance online, from the last few steps alone, and to set the momentum's sign and size from the estimate. The estimate is a two-dimensional secant fit: it compares how gradients changed against how parameters changed, which recovers the dominant rotating mode of the local Jacobian without any Hessian-vector products. We then ask a question that local theory cannot answer. Over many random initializations, does tracking the rotation enlarge the set of starting points from which training actually reaches a Nash equilibrium? Measuring that set directly is how we turn the "global convergence gap" into a number.
Core Claim
The optimal momentum coefficient for a game depends on the angle of the Jacobian's eigenvalues in the complex plane. Write such an eigenvalue as $\lambda = \alpha + i\omega$: the real part $\alpha$ pulls the iterates toward the solution and the imaginary part $\omega$ makes them rotate. Far from equilibrium this angle varies along the trajectory, so any fixed coefficient is wrong for part of the run. It over-damps in regions where descent dominates and under-damps in regions where rotation dominates. Our claim is that a cheap secant estimate of the local angle, fed into a precomputed table of optimal heavy-ball coefficients, produces two effects. First, it enlarges the fraction of random initializations that converge to a Nash equilibrium on low-dimensional nonconvex games. Second, it matches or beats the best hand-tuned static momentum on GAN training at equal tuning budget. A side effect is that the sign of momentum is chosen automatically, which gives a concrete, testable reading of the apparent sign contradiction in the literature.
Where This Sits
Three results motivate the proposal. Negative Momentum for Improved Game Dynamics proves that alternating updates with a fixed negative coefficient converge linearly on bilinear games, and Convergence of Gradient Methods on Bilinear Zero-Sum Games shows that at least one heavy-ball coefficient must be negative there. In contrast, When is Momentum Extragradient Optimal? proves that positive momentum is optimal for cross-shaped and shifted-imaginary spectra when combined with extragradient. Taken together, these suggest that the right sign is a function of spectral geometry, not a universal constant. All three analyses are local or affine, and all fix the coefficient in advance.
Complex Momentum for Optimization in Games is the closest existing method. It makes the coefficient complex, so the momentum buffer counter-rotates, and it reports a sweet spot near phase $\pi/8$. That phase is still a single hand-tuned constant. We keep momentum real but let its value follow the measured local angle, and we compare against complex momentum directly.
Three results constrain the design.
- Negative momentum alone is suboptimal near equilibrium. On the Suboptimality of Negative Momentum for Minimax Optimization proves a local ceiling of $\mathcal{O}(\kappa^{1.5})$ iterations, where $\kappa$ is the game's condition number, against $\mathcal{O}(\kappa)$ for extragradient. We therefore do not claim to beat extragradient's local rate. Our claim concerns reaching the basin of attraction in the first place, and per-gradient efficiency.
- Some spectral shapes admit no acceleration. Accelerating Smooth Games by Manipulating Spectral Shapes shows that for disc-shaped spectra, no oblivious first-order method beats plain gradient descent. Any table we build must fall back to zero momentum in that regime rather than pretend otherwise.
- The spectrum can mislead. The non-normality problem noted for this field (transient blow-ups even when all eigenvalues are stable) means a two-dimensional estimate can be fooled. We test for this explicitly.
Two further papers supply ingredients. The continuous-time analyses in Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games and Understanding Dynamics of Adam in Zero-Sum Games show that lower or negative momentum widens the stable step-size window. They also give the strongest practical static setting: Adam with first-moment coefficient $-0.3$. The secant idea itself comes from quasi-Newton approximation of Jacobians for variational inequalities (Exploring Jacobian Inexactness in Second-Order Methods). We use it only to read off an angle, not to precondition.
The Method
At step $t$ we already hold the last two parameter steps $\Delta_{t-1}, \Delta_t$ and the corresponding gradient changes $y_{t-1}, y_t$. Here $y_t = v(\theta_{t+1}) - v(\theta_t)$ is the change in the stacked gradient of both players. Because $y \approx \mathcal{J}\Delta$ locally, projecting the Jacobian onto the span of recent steps gives a $2\times 2$ estimate:
$$ \hat H_t = (S_t^\top S_t)^{-1} S_t^\top Y_t, \qquad S_t = [\Delta_{t-1}, \Delta_t],\; Y_t = [y_{t-1}, y_t]. $$
The eigenvalues of $\hat H_t$ are a real pair or a conjugate pair $\hat\alpha \pm i\hat\omega$. The quantity we act on is the angle $\hat\phi = \arctan(|\hat\omega|/\hat\alpha)$. We smooth the Gram matrices $S^\top S$ and $S^\top Y$ with an exponential moving average to suppress minibatch noise, and we track a running ratio $\hat r$ of the largest to smallest estimated magnitudes. Each step costs a handful of inner products and two stored vectors; no double backpropagation is needed. For Adam, we apply the same fit to preconditioned gradients, so the estimate refers to the operator Adam actually iterates.
The coefficient comes from a table computed once, offline. For heavy ball applied to the linear field $\lambda x$, the per-mode characteristic polynomial is $z^2 - (1+\beta-\eta\lambda)z + \beta$, where $\eta$ is the step size and $\beta$ the momentum coefficient. For each angle $\phi$ and ratio $r$, we numerically find the pair $(\eta, \beta)$ that minimizes the worst root modulus over the sector $\{|\arg\lambda|\le\phi,\ 1\le|\lambda|\le r\}$:
$$ \beta^*(\phi, r) = \arg\min_{\beta}\ \min_{\eta}\ \max_{\lambda \in K(\phi,r)} \rho\big(z^2 - (1+\beta-\eta\lambda)z + \beta\big), $$
where $K(\phi, r)$ is that sector and $\rho$ denotes the largest root modulus. By construction $\beta^*$ is positive near $\phi = 0$ (the classical Polyak regime), crosses zero at some intermediate angle, and becomes negative as $\phi$ approaches $\pi/2$. We expect a disc-like sector to drive it to roughly zero, consistent with the spectral-shapes result. During training we look up $\beta^*(\hat\phi_t, \hat r_t)$, rescale the step by the estimated magnitude, and limit how fast $\beta$ may change between steps. If $\hat\alpha < 0$ (the local field repels), we set $\beta = 0$ and log the event, since momentum of either sign cannot fix repulsion.
Research Program
The work has three parts.
Estimator validity. On bilinear and quadratic games with planted spectra (cross, thin ellipse, disc, shifted imaginary), we check how well $\hat\phi$ tracks the true dominant angle under deterministic and noisy gradients. On deliberately non-normal Jacobians with the same spectrum, we check whether transient amplification fools it.
Basin measurement. We treat the global convergence gap as a quantity: the fraction of a dense grid of initializations that reaches a Nash equilibrium within a fixed gradient budget. We record separately the fraction that ends in limit cycles, the fraction that ends at stable points which are not Nash (the failure mode identified in Finding and Only Finding Differential Nash Equilibria), and the fraction that diverges. The test problems are small enough to enumerate:
- Dirac-GAN with a nonlinear discriminator;
- two- and four-dimensional polynomial games with known limit cycles and known non-Nash attractors;
- the 8-Gaussian ring with small multilayer perceptrons.
Theory. We pursue a restricted global statement. Consider strongly monotone games whose Jacobian angle varies across space but stays below $\pi/2$ (so every region still contracts). For such games, any fixed $\beta$ has a global rate set by the worst region. A tracker with bounded lag should achieve a rate governed by the regions it actually visits. We will attempt an energy-function proof under a slowly varying Jacobian assumption, and we state in advance that the nonconvex case will remain empirical.
Baselines
The naive reference points are alternating gradient descent-ascent with no momentum and Adam with its common GAN setting $\beta_1 = 0.5$. On a CIFAR-10 ResNet GAN, Understanding Dynamics of Adam in Zero-Sum Games reports an Inception Score of about 4.16 at $\beta_1 = 0.5$.
The strongest competitor on the core question is the best static coefficient. We run it in two forms:
- real negative momentum on Adam, which the same paper reports raises the Inception Score to about 7.08 at $\beta_1 = -0.3$;
- complex momentum with tuned phase from Complex Momentum for Optimization in Games, which reports an Inception Score of 9.25 against 9.10 on BigGAN.
These are the right opponents because our claim is precisely that a fixed coefficient, however well chosen, is the bottleneck. For fairness, each static method receives the same number of tuning trials as our method's few hyperparameters (moving-average decay and rate limit), on the same architecture, data, seeds, and gradient budget. Beating them means a statistically clear gain in Inception Score or FID at equal trials on CIFAR-10, and a strictly larger Nash-reaching basin fraction on the small games, using the best static $\beta$ chosen per game with oracle access.
Extragradient and optimistic gradient descent-ascent represent the locally optimal methods. We compare them per gradient evaluation, since extragradient spends two gradients per step. Momentum extragradient with positive momentum, from When is Momentum Extragradient Optimal?, is the representative of the opposite sign convention.
At scale, the reference stabilizers are Lookahead-Minmax (Taming GANs with Lookahead-Minmax), which reports FID 12.19 on CIFAR-10, and two-timescale updates (Local Convergence Analysis of GDA with Finite Timescale Separation), which reports CelebA FID 8.0 at $\tau = 8$ against 13.1 at $\tau = 1$. We wrap our optimizer inside Lookahead as well, because the two are complementary rather than exclusive.
Two ablations isolate the key ingredient. The first replaces the two-dimensional secant fit with a scalar heuristic: the sign of the inner product of consecutive updates. This tests whether separating rotation (successive steps turn by roughly 90 degrees) from plain overshoot (they flip by roughly 180 degrees) matters. The second freezes $\beta$ at the trajectory-averaged value our tracker chose, which tests whether varying the coefficient matters or only its typical level.
Why not just…?
Why not just grid-search a fixed negative momentum? This is the most serious threat. If the angle along realistic GAN trajectories barely moves, a well-tuned constant will match us, and the idea reduces to an automatic tuner. The evidence that it might not suffice is that the best static Adam setting reported on Lacuna (Inception Score about 7.08) was found by sweeping and has no claim of optimality. Even so, we treat this as the main question rather than something to argue past. The frozen-average ablation and the oracle per-game static baseline settle it directly. We also log the measured angle across training: if its spread is small, we will report that adaptivity is unnecessary.
Why not just use extragradient, which is locally optimal? Extragradient's $\mathcal{O}(\kappa)$ rate is a local statement and costs two gradients per step. It also converges to stable points that are not Nash equilibria, per Finding and Only Finding Differential Nash Equilibria. The basin measurements report extragradient's Nash-reaching fraction at equal gradient budget. If extragradient dominates there too, our method's remaining value is the single-gradient cost, and we will say so plainly.
Why not just use complex momentum, which already counter-rotates? Complex momentum fixes one phase for the whole run. Its reported gain over the BigGAN baseline is modest (Inception Score 9.25 against 9.10), so it is far from clear that one phase is right everywhere. A tracker is also not exclusive to complex momentum: as a secondary arm, the estimated angle could set the complex phase instead of a real $\beta$. Running both arms shows whether the gain comes from complex buffers or from tracking.
Why not just separate timescales? The threshold that guarantees convergence for two-timescale updates is intractable to compute for deep networks, and the guarantee is local. We include $\tau \in \{1, 4, 8\}$ in the sweeps and report whether tracking still helps once $\tau$ is tuned.
Proposed Experiments
The first decisive check is the estimator. On planted-spectrum quadratic games of dimension 100 to 1000, matching the bilinear sizes in Accelerating Smooth Games by Manipulating Spectral Shapes, the smoothed angle should land within a few degrees of the dominant true angle. On non-normal variants we report how often it errs. If it fails here, the project stops.
The second check is basin fraction on Dirac-GAN, the polynomial games, and the 8-Gaussian ring. We report Nash-reaching, cycling, non-Nash, and diverged fractions over a dense initialization grid, against every baseline. The numbers we want are a larger Nash fraction than the oracle-tuned static coefficient, and no increase in the non-Nash fraction. No reported basin-fraction result was found on Lacuna for any of these problems, so all these numbers are new.
The third check is scale. We train the CIFAR-10 ResNet GAN setup from the Adam ODE paper, plus CelebA, with Inception Score and FID over at least five seeds and equal tuning trials, against the reported reference points of about 7.08 Inception Score (static negative Adam), 12.19 FID (Lookahead-Minmax), and 8.0 CelebA FID (two-timescale updates). We also log wall-clock overhead and the trajectory of the chosen $\beta$, including how often its sign flips.
The fourth check tests the sign-reconciliation hypothesis. If sign is a function of geometry, then on spectra designed to be cross-shaped, the tracker's chosen sign under single-gradient heavy ball should differ systematically from the sign that is optimal once an extragradient step is added. We test this by recomputing the table for momentum extragradient and checking whether the zero crossing moves.
References
- Negative Momentum for Improved Game Dynamics
- Convergence of Gradient Methods on Bilinear Zero-Sum Games
- When is Momentum Extragradient Optimal? A Polynomial-Based Analysis
- Complex Momentum for Optimization in Games
- On the Suboptimality of Negative Momentum for Minimax Optimization
- Accelerating Smooth Games by Manipulating Spectral Shapes
- Continuous-Time Analysis of Heavy Ball Momentum in Min-Max Games
- Understanding Dynamics of Adam in Zero-Sum Games: An ODE Approach
- Exploring Jacobian Inexactness in Second-Order Methods for Variational Inequalities: Lower Bounds, Optimal Algorithms and Quasi-Newton Approximations
- Finding and Only Finding Differential Nash Equilibria by Both Pretending to be a Follower
- Taming GANs with Lookahead-Minmax
- Local Convergence Analysis of Gradient Descent Ascent with Finite Timescale Separation