In optimization theory, there is a long-standing "averaging trick" used to prove that algorithms work. If you want to show that Stochastic Gradient Descent (SGD) converges, it is mathematically much easier to prove that the average of all the points the algorithm visited is close to the optimum.
However, almost no one actually uses the average in practice. Deep learning engineers, researchers training GANs, and practitioners using ADMM almost exclusively use the last iterate—the very final set of weights produced by the training run. This creates a significant gap between the "ergodic" (averaged) guarantees provided by theorists and the "non-ergodic" (last-state) results used in the real world.
The Mathematical Necessity of Averaging
The preference for averaging in theory stems from its ability to "cancel out" noise and oscillations. In Anytime Online-to-Batch, Optimism and Acceleration, Ashok Cutkosky explains that standard online-to-batch conversions rely on this averaging to turn regret bounds into convergence rates. Without it, the individual points an algorithm visits might be highly suboptimal, even if their mean is perfect.
This discrepancy was famously highlighted by Ohad Shamir in the Open Problem: Is Averaging Needed for Strongly Convex Stochastic Gradient Descent?. While we know the squared distance to the optimum for the last iterate of SGD decreases at an $O(1/T)$ rate, proving the same for the function value in non-smooth settings remained an elusive challenge for years.
Why Practitioners Avoid Averaging
If averaging makes the math work, why don't we use it? Damek Davis and Wotao Yin point out in Convergence rate analysis of several splitting schemes that averaging often destroys structural properties like sparsity. If your last iterate is sparse because of a proximal operator, the average of all your iterates likely won't be.
Furthermore, in large-scale settings, storing a running average of billions of parameters is memory-intensive. In many cases, the last iterate simply performs better empirically, leading researchers like Othmane Sebbouh to investigate why practical "hacks" like avoiding averaging in SVRG are actually theoretically optimal.
Solving the Saddle Point Problem
The stakes for last-iterate convergence are highest in min-max optimization, such as training Generative Adversarial Networks (GANs). In these games, standard gradient descent-ascent often circles the equilibrium without ever reaching it. While the average might converge to the center of the circle, the last iterate stays on the perimeter.
To fix this, researchers have turned to more sophisticated methods:
- Extragradient (EG) and Optimistic Gradient (OGDA): Aryan Mokhtari, Asuman E. Ozdaglar, and Sarath Pattathil showed that these methods act as approximations of the Proximal Point Method, which naturally pulls the last iterate toward the solution (link).
- Tight Rates: Yang Cai and Weiqiang Zheng used computer-aided Sum-of-Squares (SOS) programming to prove a tight $O(1/\sqrt{T})$ last-iterate rate for EG in constrained settings (link).
- Anchoring: Ernest K. Ryu and Wotao Yin introduced "anchored" gradients to stabilize the last iterate in stochastic minimax problems (link).
New Frameworks for Non-Ergodic Analysis
Proving that the last iterate converges requires different mathematical tools than the ones used for averages. Instead of looking at the sum of losses, researchers look at how a "distance to the truth" decreases over time.
- Modified Fejér Sequences: Junhong Lin, Lorenzo Rosasco, and Silvia Villa developed a framework based on fixed-point theory to show that the last iterate of splitting methods converges at the same rate as the best-found point (link).
- Lyapunov Functions: Tao Sun and colleagues used Lyapunov functions inspired by dynamical systems to prove non-ergodic $O(1/k)$ rates for inertial methods (link).
- Nesterov Extrapolation in ADMM: Huan Li and Zhouchen Lin bridged the gap for constrained problems by integrating Nesterov’s momentum into ADMM, achieving the first optimal $O(1/K)$ last-iterate rate for that class of problems (link).
Optimal Schedules for the Last Iterate
The shift toward last-iterate analysis is also changing how we think about hyperparameters. In Optimal Linear Decay Learning Rate Schedules and Further Refinements, Harsh Mehta, Konstantin Mishchenko, and Aaron Defazio argue that because theory has focused on averages, it has recommended sub-optimal learning rate schedules. When you optimize specifically for the last iterate, simple linear decay schedules often emerge as theoretically optimal, matching the "vibes-based" heuristics used by practitioners for years.
By moving away from the mathematical convenience of averaging, this research direction is finally providing formal guarantees for the models that are actually deployed in production.