TL;DR
Adaptive optimizers like Adam are prized for their speed and ease of use, but they often fail to match the final generalization performance of Stochastic Gradient Descent (SGD). This paper introduces SWATS (SWitching from Adam To Sgd), a simple, hyperparameter-free strategy that starts training with Adam and automatically transitions to SGD at the optimal moment, effectively closing the generalization gap.
What Problem Does This Paper Address?
In the deep learning community, there is a well-known trade-off between training speed and generalization. Adaptive methods like Adam, Adagrad, and RMSprop converge quickly and are relatively insensitive to learning rate tuning. However, empirical evidence suggests they often get stuck in "bad" basins of attraction, leading to worse test-set performance compared to carefully tuned SGD.
Researchers often find themselves in a dilemma: use Adam for fast prototyping, or spend days tuning SGD for peak performance. While some practitioners manually switch from Adam to SGD, this requires "babysitting" the training process to decide when to switch and what learning rate to use for the SGD phase. SWATS aims to automate this transition without adding new hyperparameters.
Methods
The core of SWATS is a triggering condition and a learning rate estimation technique.
- Estimating the SGD Learning Rate ($\gamma_k$): To ensure a smooth transition, the algorithm needs a scalar learning rate for SGD that mimics the behavior of the Adam step. The authors propose solving a projection problem: find a $\gamma_k$ such that the projection of the Adam step onto the current gradient direction is the Adam step itself. Geometry of the projection
- The Triggering Condition: The algorithm maintains an exponential moving average of this estimated $\gamma_k$. The switch to SGD is triggered when this monitored quantity stabilizes—specifically, when the difference between the bias-corrected average and the current estimate falls below a small threshold ($\epsilon$).
- Zero Overhead: By reusing Adam’s existing parameters (like $\beta_2$) for the moving average, SWATS introduces no new hyperparameters, making it a "drop-in" replacement for standard Adam.
Datasets
The authors evaluated SWATS across a diverse set of tasks:
- Image Classification: CIFAR-10 and CIFAR-100 using architectures like ResNet-32, DenseNet, PyramidNet, and SENet.
- Large-scale Vision: Tiny-ImageNet using ResNet-18.
- Language Modeling: Penn Treebank (PTB) and WikiText-2 (WT2) using AWD-LSTM and AWD-QRNN architectures.
Key Insights and Ideas
- The Generalization Gap is Real: As shown in Figure 1, standard Adam can leave a ~2% performance gap on the table compared to SGD on tasks like DenseNet for CIFAR-10. Generalization gap in DenseNet
- Timing Matters: Switching too early makes the optimizer behave like pure SGD (losing Adam's initial speed), while switching too late means the model has already settled into a poor basin of attraction from which SGD cannot recover.
- Automatic Tuning Works: The most surprising insight is that the simple projection-based estimate for the SGD learning rate ($\Lambda$) consistently converges to values very close to what a human researcher would find through exhaustive grid search.
Key Results
- Closing the Gap: On vision tasks, SWATS consistently matched the superior generalization of SGD while retaining Adam's rapid initial progress.
- Robustness: In language modeling, where Adam is traditionally very strong, SWATS performed comparably to Adam, showing it doesn't hurt performance even when the "switch" isn't strictly necessary for generalization.
- Recovery from Dips: On Tiny-ImageNet, the authors noted a curious "temporary degradation" immediately after the switch, where accuracy dropped significantly before recovering to surpass Adam's final performance. Tiny-ImageNet performance
Limitations and Future Work
- Hard vs. Soft Switch: The current implementation uses a "hard" switch, which causes the performance spikes mentioned above. The authors suggest that a "soft" transition—using a convex combination of Adam and SGD steps—might be more stable.
- Bidirectional Switching: While SWATS only moves from Adam to SGD, the authors mention that switching back and forth as needed could be a viable research direction.
- Integration with other variants: The paper focuses on standard Adam, but the logic could potentially be applied to other adaptive variants like AMSGrad or combined with techniques like cosine annealing.
Summary updated