The Theory-Practice Disconnect
When a machine learning engineer finishes training a neural network, they save the final checkpoint—the last iterate ($x_T$)—and deploy it to production. They do not deploy an average of every model state generated during the training process.
Yet, for decades, classical optimization theory told a completely different story. To prove that Stochastic Gradient Descent (SGD) achieves optimal convergence rates on convex objectives, theorists traditionally relied on iterate averaging (or ergodic convergence). Instead of analyzing the final state $x_T$, they analyzed the running average:
$$\bar{x}_T = \frac{1}{T} \sum_{t=1}^T x_t$$
As highlighted by Ruinan Jin et al., this creates a significant gap between theory and practice. If mathematical guarantees only apply to $\bar{x}_T$, we lack formal assurances for the actual models used in real-world deployment. This disconnect spans across machine learning paradigms, from online learning (Yiming Ying et al.) and momentum methods (Xiaoyu Li et al.) to reinforcement learning actor-critic frameworks (Mudit Gaur et al.) and sampling algorithms (Jelena Diakonikolas et al.).
Why the Last Iterate is Mathematically Stubborn
To understand why this gap exists, we have to look at how stochastic noise behaves during optimization.
When minimizing a loss function using stochastic gradients, each step contains a true gradient plus some zero-mean random noise: $g_t = \nabla f(x_t) + \xi_t$.
Iterate averaging is popular in theory because it acts as a low-pass filter. When you sum up $x_t$ over time, the random fluctuations $\xi_t$ tend to cancel each other out. This smoothing effect makes it straightforward to prove that $\bar{x}_T$ converges at the theoretically optimal rate of $\mathcal{O}(1/T)$ for strongly convex problems.
The last iterate $x_T$, however, enjoys no such protection. It is the product of the very last sequence of updates, meaning it is fully exposed to the variance of the final stochastic gradient steps. If the learning rate (step size) is too large, the last iterate will endlessly bounce around the optimum rather than settling into it.
Historically, analyzing the last iterate of standard SGD with a standard polynomial step size (like $\eta_t \propto 1/t$) hit a mathematical wall: the proven convergence rate suffered from a logarithmic bottleneck ($\mathcal{O}(\log T / T)$), making it look strictly worse than the averaged version. As Alexander Rakhlin et al. noted, this created a paradox where the final model performed exceptionally well in practice, but the math could only guarantee a suboptimal rate.
Closing the Gap with Suffix Averaging and Smart Schedules
To reconcile this discrepancy, researchers began looking for ways to achieve optimal convergence rates specifically for the final stages of training without forcing a full historical average.
Suffix Averaging
One of the earliest elegant fixes came from Alexander Rakhlin et al., who demonstrated that suffix averaging—averaging only a small fraction of the final iterations (e.g., the last $10\%$ of steps)—completely removes the logarithmic bottleneck. This simple modification recovers the optimal $\mathcal{O}(1/T)$ rate while remaining practically identical to deploying the final model state.
Information-Theoretic Step-Size Schedules
Another approach is to alter the learning rate schedule so that the last iterate stabilizes naturally. Prateek Jain et al. introduced novel step-size sequences that make the last iterate of SGD information-theoretically optimal, matching the speed limits of optimization without any averaging at all.
In the context of overparameterized models, single-author work by Jingfeng Wu demonstrated that a geometrically decaying step-size schedule allows the last iterate to generalize just as well as complex averaging techniques, proving that geometric decay is fundamentally superior to polynomial decay on an instance-by-instance basis.
Explaining Modern Deep Learning Schedules
This theoretical line of work directly explains why empirical tricks used by practitioners work so well. For instance, large-scale model training often uses a "Warmup-Stable-Decay" (WSD) schedule. Fabian Schaipp et al. proved that the final "cooldown" or decay phase of these schedules is exactly what mathematically eliminates the logarithmic error terms of the last iterate, providing a rigorous justification for empirical learning rate practices.
Extending to Momentum, Shuffling, and Games
Once the basic mechanisms for last-iterate convergence in standard SGD were understood, researchers extended these insights to more complex, modern optimization workflows.
- Momentum and Adaptive Methods: Momentum methods like Polyak's Heavy-ball or Nesterov acceleration are notoriously difficult to analyze for individual iterates. Wei Tao et al. proved that adaptive Heavy-ball methods achieve optimal individual convergence for the last iterate, while Ruinan Jin et al. established a comprehensive framework proving that Adam's last iterate converges under relaxed, realistic conditions similar to SGD.
- Data Shuffling: Practical training rarely samples data with replacement; instead, it shuffles the dataset at the start of each epoch. Zijian Liu et al. and Kwangjun Ahn et al. bridged this gap by proving that shuffling gradient methods guarantee strong last-iterate convergence even without assuming strong convexity.
- Min-Max Optimization and Games: In adversarial training and GANs, algorithms often cycle or diverge instead of settling. To address this, Kenshi Abe et al. developed algorithms that guarantee last-iterate convergence to the Nash equilibrium under noisy feedback, ensuring that the final trained model is stable and optimal.
By shifting the analytical focus from historical averages to the final state, this research direction has successfully closed the theory-practice gap, ensuring that mathematical guarantees finally match the models actually deployed in production.
Key Foundational Work
Open Problem: Is Averaging Needed for Strongly Convex Stochastic Gradient Descent? (2012) — contributed 3 concept elements to this direction.
Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization (2012) — contributed 2 concept elements to this direction.