Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond

Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, Shuicheng Yan

Deep learning optimization relies heavily on Stochastic Gradient Descent (SGD) and its variants. However, SGD inherently struggles with a "tug-of-war" between optimization (minimizing training loss) and generalization (performing well on unseen data). If an optimizer is too aggressive, it risks overfitting; if it is too cautious, it fails to find a good solution.

In 2019, the Lookahead optimizer emerged as a powerful alternative. By maintaining two sets of weights—"fast weights" for exploration and "slow weights" that trail behind—Lookahead consistently achieves better test performance than vanilla SGD. Yet, the mathematical reasons behind this "generalization gap" have remained elusive. This paper provides the first formal proof explaining why Lookahead balances this trade-off so effectively and introduces a novel, even more powerful variant: Stagewise Locally-Regularized LookAhead (SLRLA).

The Theoretical Gap & Excess Risk Error

To understand Lookahead's superiority, the researchers analyzed the Excess Risk Error, a metric that combines both optimization error and generalization error.

Their theoretical analysis proves that by carefully tuning the "slow weight" step size (denoted as $\alpha$), Lookahead achieves a lower total error than SGD. This holds true for convex problems and specific non-convex problems that satisfy the Polyak-Łojasiewicz (PŁ) condition.

Introducing SLRLA

Building on these theoretical foundations, the authors propose SLRLA. This algorithm enhances the standard Lookahead approach in two key ways:

  1. Stagewise Training: The training process is divided into distinct stages, with the learning rate decaying at the start of each new stage.
  2. Local Regularization: At each stage, a local penalty is added to the loss function. This regularizer discourages the weights from drifting too far from the stage's starting point.

Effectively, this local penalty "smooths" the optimization landscape. It makes the problem behave more like a well-conditioned convex optimization task, accelerating convergence while preventing the model from settling into "sharp" minima that generalize poorly.

Key Insights: Why It Works

  • The $\alpha$ Balancing Act: The slow weight update acts as a variance reduction mechanism. While setting $\alpha=1$ (equivalent to vanilla SGD) might optimize the training loss faster, a smaller $\alpha$ significantly reduces generalization error, hitting a "sweet spot" for test performance.
  • The PŁ Condition in Deep Networks: The paper confirms that deep neural networks frequently satisfy the PŁ condition—where the gradient grows at least as fast as the square root of the loss—along their optimization paths. This validates the application of their theoretical proofs to real-world, non-convex deep learning models.
  • Regularization as a Catalyst: Surprisingly, the local regularizer in SLRLA does more than just prevent overfitting. It enables linear convergence (exceptionally fast optimization) on non-convex problems, provided the landscape isn't excessively chaotic (i.e., the Hessian's smallest eigenvalue isn't too negative).

Empirical Results

The researchers validated SLRLA across standard computer vision benchmarks, including MNIST, CIFAR-10, CIFAR-100, and ImageNet, using architectures like ResNet18, VGG16, and Wide-ResNet-16-10. SLRLA consistently outperformed both standard SGD and the original Stagewise Lookahead (SLA):

  • CIFAR-100 (ResNet18): SLRLA achieved 78.58% accuracy, beating SLA (78.34%) and Stagewise SGD (78.24%).
  • ImageNet (ResNet18): SLRLA reached a 70.47% top-1 accuracy, a notable improvement over the 70.30% from standard Lookahead.
  • Stability: SLRLA proved highly robust. Even when the regularization constant ($\beta$) varied widely, the model maintained high performance, simplifying the hyperparameter tuning process for practitioners.

Limitations and Future Directions

The primary limitation of this work is its reliance on the PŁ condition or weak-quasi-convexity for theoretical guarantees. While empirically true for networks like ResNet, these are not universal properties of all non-convex functions.

Future research could explore whether SLRLA's benefits extend to more complex architectures like Transformers or the highly jagged optimization landscapes found in Reinforcement Learning. Additionally, developing methods to automatically tune the local regularization parameter ($\beta$) during training remains an exciting open challenge.

Summary updated

Abstract

To train networks, lookahead algorithm~\cite{zhang2019lookahead} updates its fast weights $k$ times via an inner-loop optimizer before updating its slow weights once by using the latest fast weights. Any optimizer, e.g. SGD, can serve as the inner-loop optimizer, and the derived lookahead generally enjoys remarkable test performance improvement over the vanilla optimizer. But theoretical understandings on the test performance improvement of lookahead remain absent yet. To solve this issue, we theoretically justify the advantages of lookahead in terms of the excess risk error which measures the test performance. Specifically, we prove that lookahead using SGD as its inner-loop optimizer can better balance the optimization error and generalization error to achieve smaller excess risk error than vanilla SGD on (strongly) convex problems and nonconvex problems with Polyak-{\L}ojasiewicz condition which has been observed/proved in neural networks. Moreover, we show the stagewise optimization strategy~\cite{barshan2015stage} which decays learning rate several times during training can also benefit lookahead in improving its optimization and generalization errors on strongly convex problems. Finally, we propose a stagewise locally-regularized lookahead (SLRLA) algorithm which sums up the vanilla objective and a local regularizer to minimize at each stage and provably enjoys optimization and generalization improvement over the conventional (stagewise) lookahead. Experimental results on CIFAR10/100 and ImageNet testify its advantages. Codes is available at \url{https://github.com/sail-sg/SLRLA-optimizer}.
BibTeX
@inproceedings{zhou2021towards,
  author = {Pan Zhou and Hanshu Yan and Xiaotong Yuan and Jiashi Feng and Shuicheng Yan},
  title = {{Towards Understanding Why Lookahead Generalizes Better Than SGD and Beyond}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure