Improving Generalization Performance by Switching from Adam to SGD

Nitish Shirish Keskar, Richard SocherarXiv 2017 · arXiv

TL;DR

Adaptive optimizers like Adam are prized for their speed and ease of use, but they often fail to match the final generalization performance of Stochastic Gradient Descent (SGD). This paper introduces SWATS (SWitching from Adam To Sgd), a simple, hyperparameter-free strategy that starts training with Adam and automatically transitions to SGD at the optimal moment, effectively closing the generalization gap.

What Problem Does This Paper Address?

In the deep learning community, there is a well-known trade-off between training speed and generalization. Adaptive methods like Adam, Adagrad, and RMSprop converge quickly and are relatively insensitive to learning rate tuning. However, empirical evidence suggests they often get stuck in "bad" basins of attraction, leading to worse test-set performance compared to carefully tuned SGD.

Researchers often find themselves in a dilemma: use Adam for fast prototyping, or spend days tuning SGD for peak performance. While some practitioners manually switch from Adam to SGD, this requires "babysitting" the training process to decide when to switch and what learning rate to use for the SGD phase. SWATS aims to automate this transition without adding new hyperparameters.

Methods

The core of SWATS is a triggering condition and a learning rate estimation technique.

  1. Estimating the SGD Learning Rate ($\gamma_k$): To ensure a smooth transition, the algorithm needs a scalar learning rate for SGD that mimics the behavior of the Adam step. The authors propose solving a projection problem: find a $\gamma_k$ such that the projection of the Adam step onto the current gradient direction is the Adam step itself. Geometry of the projection
  2. The Triggering Condition: The algorithm maintains an exponential moving average of this estimated $\gamma_k$. The switch to SGD is triggered when this monitored quantity stabilizes—specifically, when the difference between the bias-corrected average and the current estimate falls below a small threshold ($\epsilon$).
  3. Zero Overhead: By reusing Adam’s existing parameters (like $\beta_2$) for the moving average, SWATS introduces no new hyperparameters, making it a "drop-in" replacement for standard Adam.
Figure 3. Illustrating the learning rate for SGD (γk) estimated by our proposed projection given an iterate wk, a stochastic gradient gk and the Adam step pk.

Datasets

The authors evaluated SWATS across a diverse set of tasks:

  • Image Classification: CIFAR-10 and CIFAR-100 using architectures like ResNet-32, DenseNet, PyramidNet, and SENet.
  • Large-scale Vision: Tiny-ImageNet using ResNet-18.
  • Language Modeling: Penn Treebank (PTB) and WikiText-2 (WT2) using AWD-LSTM and AWD-QRNN architectures.

Key Insights and Ideas

  • The Generalization Gap is Real: As shown in Figure 1, standard Adam can leave a ~2% performance gap on the table compared to SGD on tasks like DenseNet for CIFAR-10. Generalization gap in DenseNet
  • Timing Matters: Switching too early makes the optimizer behave like pure SGD (losing Adam's initial speed), while switching too late means the model has already settled into a poor basin of attraction from which SGD cannot recover.
  • Automatic Tuning Works: The most surprising insight is that the simple projection-based estimate for the SGD learning rate ($\Lambda$) consistently converges to values very close to what a human researcher would find through exhaustive grid search.
Figure 1. Training the DenseNet architecture on the CIFAR-10 data set with four optimizers: SGD, Adam, Adam-Clip(1, ∞) and Adam-Clip(0, 1). SGD achieves the best testing accuracy while training with Adam leads to a generalization gap of roughly 2%. Setting a minimum learning rate for each parameter of Adam par- tially closes the generalization gap.

Key Results

  • Closing the Gap: On vision tasks, SWATS consistently matched the superior generalization of SGD while retaining Adam's rapid initial progress.
  • Robustness: In language modeling, where Adam is traditionally very strong, SWATS performed comparably to Adam, showing it doesn't hurt performance even when the "switch" isn't strictly necessary for generalization.
  • Recovery from Dips: On Tiny-ImageNet, the authors noted a curious "temporary degradation" immediately after the switch, where accuracy dropped significantly before recovering to surpass Adam's final performance. Tiny-ImageNet performance
Tiny-ImageNet performance

Limitations and Future Work

  • Hard vs. Soft Switch: The current implementation uses a "hard" switch, which causes the performance spikes mentioned above. The authors suggest that a "soft" transition—using a convex combination of Adam and SGD steps—might be more stable.
  • Bidirectional Switching: While SWATS only moves from Adam to SGD, the authors mention that switching back and forth as needed could be a viable research direction.
  • Integration with other variants: The paper focuses on standard Adam, but the logic could potentially be applied to other adaptive variants like AMSGrad or combined with techniques like cosine annealing.

Summary updated

Abstract

Despite superior training outcomes, adaptive optimization methods such as Adam, Adagrad or RMSprop have been found to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well in the initial portion of training but are outperformed by SGD at later stages of training. We investigate a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate. Concretely, we propose SWATS, a simple strategy which switches from Adam to SGD when a triggering condition is satisfied. The condition we propose relates to the projection of Adam steps on the gradient subspace. By design, the monitoring process for this condition adds very little overhead and does not increase the number of hyperparameters in the optimizer. We report experiments on several standard benchmarks such as: ResNet, SENet, DenseNet and PyramidNet for the CIFAR-10 and CIFAR-100 data sets, ResNet on the tiny-ImageNet data set and language modeling with recurrent networks on the PTB and WT2 data sets. The results show that our strategy is capable of closing the generalization gap between SGD and Adam on a majority of the tasks.
BibTeX
@misc{keskar2017improving,
  author = {Nitish Shirish Keskar and Richard Socher},
  title = {{Improving Generalization Performance by Switching from Adam to SGD}},
  year = {2017},
  eprint = {1712.07628},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.07628},
}
Figure