Classic optimization theory dictates that to guarantee convergence to a stationary point or a Nash equilibrium, the learning rate must decay toward zero over time. This vanishing step size acts as an artificial dampening force, preventing the algorithm from overshooting as it nears the optimum.
However, diminishing learning rates introduce a severe practical drawback: they drastically slow down training in later stages and make learning agents unresponsive to dynamic environmental shifts. In real-world applications—from training deep neural networks to deploying high-frequency trading bots—practitioners almost exclusively favor large, constant learning rates.
This creates a profound gap between theoretical necessity and empirical success. How can an algorithm maintain a fixed update magnitude without oscillating, cycling, or spiraling out of control? Recent research bridges this gap by uncovering the mathematical mechanisms—such as predictive optimism, implicit architectural scaling, and stationary distributions—that enable constant learning rates to achieve stable, "last-iterate" convergence.
The Problem: The Rotational Momentum of Games
To understand why constant learning rates are problematic, consider a standard zero-sum matrix game optimized via Gradient Descent-Ascent (GDA). If one player minimizes and the other maximizes, the gradient field forms a vortex around the Nash equilibrium.
When we apply a constant learning rate to standard GDA or the Multiplicative Weights Update (MWU) algorithm, the discrete steps inevitably overshoot the curved vector field. Instead of spiraling inward, the iterates orbit perpetually or spiral outward, moving further away from the solution. Historically, the only way to prove convergence was to force the learning rate to decay, or to average all historical iterates together. But average-iterate convergence is a weak guarantee; the actual last iterate (the final model state) might still be far from the equilibrium.
Mechanism 1: Optimism as a Damping Force
The most prominent solution to the cycling problem is optimism. Instead of updating parameters based solely on the current gradient, an optimistic algorithm incorporates a "look-ahead" mechanism, adjusting its step based on a forecast of the next gradient.
In Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization, Daskalakis et al. proved that the Optimistic Multiplicative Weights Update (OMWU) algorithm achieves last-iterate convergence in constrained zero-sum games using a constant learning rate. Geometrically, the optimistic term acts as a negative feedback loop. By anticipating the opponent's movement, it introduces a friction-like force that counteracts the outward rotational momentum of the game, steering the iterate directly into the equilibrium.
Standard MWU: Current State ──> Overshoots Curve ──> Spirals Outward
Optimistic MWU: Current State ──> Predicts Next Move ──> Dampens Orbit ──> Converges to Equilibrium
This stabilizing effect is remarkably robust across different landscapes and game structures:
- Convex-Concave Landscapes: Lei et al. demonstrated in Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapes that constant learning rates combined with optimism achieve exact last-iterate convergence faster than traditional vanishing step-size schedules.
- Linear (Exponential) Convergence: When the objective landscape exhibits structural regularity—specifically Saddle-Point Metric Subregularity—Wei et al. showed in Linear Last-iterate Convergence in Constrained Saddle-point Optimization that optimistic algorithms like OGDA and OMWU achieve exponential convergence to the solution set without any step-size decay.
- Multi-Player and Sequential Games: This principle scales successfully to multi-player settings (Cai et al. in Finite-Time Last-Iterate Convergence for Learning in Multi-Player Games) and complex sequential games (Lee et al. in Kernelized Multiplicative Weights for 0/1-Polyhedral Games: Bridging the Gap Between Learning in Extensive-Form and Normal-Form Games), proving that agents can remain highly responsive to new information without sacrificing the stability of their final state.
A closely related predictive mechanism is explored by Wang et al. in Convergence of Multi-Agent Learning with a Finite Step Size in General-Sum Games. Their algorithm, Gradient Ascent with Shrinking Policy Prediction (GA-SPP), uses a finite, constant step size but dynamically shrinks the policy prediction step as training progresses, stabilizing general-sum multi-agent interactions.
Mechanism 2: Implicit Decay and Adaptive Scaling
In deep learning, models often converge perfectly even when the global learning rate hyperparameter is kept completely flat. This occurs because certain architectural components and adaptive optimizers perform an implicit scaling that mimics a decaying rate.
Batch Normalization as an Automatic Tuner
In Theoretical Analysis of Auto Rate-Tuning by Batch Normalization, Arora et al. analyzed the scale-invariance property of Batch Normalization (BN). Because BN normalizes activations, multiplying a layer's weights by a constant factor does not change the network's output. However, it does change the gradient magnitude.
The authors proved that as the weight vectors naturally grow in length during training, the effective learning rate automatically decays at an $O(1/t)$ rate. Thus, the architecture itself provides the necessary stabilization, allowing the model to reach stationary points under a fixed global rate.
Constant Hyperparameters in Adaptive Optimizers
Adaptive algorithms like Adam and AMSGrad scale updates based on running averages of historical gradients. Traditional proofs required these adaptive schedules or momentum parameters to decay over time to guarantee convergence.
However, Iiduka proved in Appropriate Learning Rates of Adaptive Learning Rate Optimization Algorithms for Training Deep Neural Networks that adaptive optimizers using constant learning rates are mathematically guaranteed to converge to an approximate stationary point in non-convex landscapes. The final distance to the true optimizer is strictly bounded by the magnitude of the step size and momentum.
To bridge the gap further, Alacaoglu et al. analyzed constrained, weakly convex stochastic optimization in Convergence of adaptive algorithms for constrained weakly convex optimization. They showed that AMSGrad achieves near-optimal convergence rates while keeping momentum and hyperparameters completely constant. In a companion work, A new regret analysis for Adam-type algorithms, the authors established that a constant first-order momentum parameter ($\beta_1$) is theoretically optimal, eliminating the need for complex time-varying schedules in practical implementations.
Mechanism 3: Statistical and Ergodic Convergence
When optimization occurs in highly stochastic or non-convex environments, a constant learning rate prevents the iterate from ever settling into a single, static point. Instead of viewing this as a failure, modern analysis redefines what it means to "converge."
Convergence to a Stationary Distribution
In high-dimensional and overparameterized settings, Stochastic Gradient Descent (SGD) with a constant learning rate does not stop; it bounces around the optimal region indefinitely due to persistent gradient noise.
In Statistical Guarantees for High-Dimensional Stochastic Gradient Descent, Li et al. proved that these constant-rate iterates converge to a stable stationary distribution rather than a single point. The distance between the mean of this distribution and the true optimizer can be rigorously bounded, providing a solid statistical foundation for using non-decaying rates in high dimensions.
Order Out of Chaos
In game-theoretic routing scenarios, aggressive constant learning rates can destabilize the system entirely, causing individual agent behaviors to become chaotic and unpredictable.
Yet, Bielawski et al. uncovered a remarkable property in Follow-the-Regularized-Leader Routes to Chaos in Routing Games. They proved that while the day-to-day trajectories of Follow-the-Regularized-Leader (FoReL) dynamics are chaotic, the system is ergodic: the long-term time average of these chaotic trajectories still converges perfectly to the exact Nash equilibrium.
Iterate Trajectory: ──> Chaotic ──> Unpredictable ──> Non-convergent
Time Average: ──> Smooths Noise ──> Filters Chaos ──> Exact Nash Equilibrium
Summary of Frameworks
The transition from decaying schedules to constant step-size guarantees spans several optimization domains, summarized below:
| Mechanism | Key Papers | Core Intuition | Target Setting |
|---|---|---|---|
| Predictive Optimism | Daskalakis et al.<br>Lei et al.<br>Wei et al. | Looking ahead introduces a negative feedback loop that dampens rotational cycling. | Min-Max Optimization & Zero-Sum Games |
| Implicit Scale Invariance | Arora et al. | Growth in weight magnitude naturally shrinks the effective step size. | Normalized Deep Networks |
| Adaptive Bounding | Iiduka<br>Alacaoglu et al. | Historical gradient scaling bounds the distance to stationary points without parameter decay. | Non-convex & Weakly Convex Optimization |
| Stationary Distributions | Li et al. | Iterates settle into a stable probability distribution centered near the true optimizer. | High-Dimensional Stochastic SGD |
By shifting the analytical focus from traditional point-convergence to structural damping, adaptive scaling, and distributional stability, these frameworks explain why constant learning rates work so well in practice—allowing fast, responsive optimization without the risk of instability.
Key Foundational Work
Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimization (2018) — contributed 3 concept elements to this direction.
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization (2019) — contributed 1 concept elements to this direction.
Multiplicative Weights Updates with Constant Step-Size in Graphical Constant-Sum Games (2018) — contributed 1 concept elements to this direction.