Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization

Difan Zou, Yuan Cao, Yuanzhi Li, Quanquan Gu

Abstract

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.

Introduction

Deep learning practitioners often face a trade-off: adaptive optimizers like Adam converge much faster than standard Stochastic Gradient Descent (SGD) but frequently result in models with higher test errors. This "generalization gap" has been widely observed in tasks like image classification. While previous research attempted to explain this gap using linear models or convex settings, these explanations often fall short. In convex regimes, adding even a small amount of weight decay regularization forces all optimizers to converge to the same unique global solution, effectively eliminating the gap. This paper investigates the hypothesis that the generalization gap is fundamentally a product of the non-convex optimization landscape of neural networks, where Adam is more prone to fitting noise patches in data that standard regularization cannot fix.

Methodology

The study employs a mix of theoretical analysis and empirical validation. The researchers analyze a two-layer convolutional neural network (CNN) trained on a specially designed "image-like" data model. In this model, each data point consists of two patches: a feature patch (correlated with the label) and a noise patch (containing random noise and a "feature noise" component inversely correlated with the label).

Training is analyzed in two stages: a "pattern learning stage," where the model picks up features or noise, and a "regularization stage," where weight decay pulls the model toward a local minimum. The behavior of Adam and GD is contrasted in both this non-convex CNN setting and a standard convex setting (linear regression). Empirical verification is conducted on the CIFAR-10 dataset using an AlexNet architecture to visualize learned filters.

Key Insights

A central finding is the differing sensitivity of optimizers to data patch properties. Adam is highly sensitive to the sparsity of data patches. In the proposed model, noise patches are less sparse than feature patches, making them "attractive" to Adam's coordinate-wise learning rate adjustment. Conversely, GD is driven by the $\ell_2$ norm of patches, which favors the true features.

Furthermore, the authors identify a "flip" phenomenon: early in training, Adam may begin to learn features but then "flips" its behavior to fit noise patches because they offer a faster path to reducing training loss in a non-convex space. Crucially, in a non-convex landscape, the "noisy" solution Adam finds is a local minimum that weight decay cannot escape, whereas in a convex setting, all optimizers would eventually converge to the same global minimum.

Key Results

The theoretical analysis yields provable failure for Adam and success for GD in the non-convex CNN setting. Adam can achieve zero training error while its test error remains no better than a random guess ($\approx 0.5$), whereas GD achieves zero training error and nearly zero test error under the same conditions.

Empirical tests on CIFAR-10 support these findings. Visualizing the first-layer filters of AlexNet reveals that Adam's filters are significantly noisier and less structured than those trained with SGD, indicating overfitting to training data noise.

Comparison of first-layer AlexNet filters trained by Adam (left) and SGD (right). Adam's filters appear significantly more noisy, indicating overfitting to training data noise.

Finally, the authors prove "Convex Equivalence": for any convex and smooth objective with positive weight decay, Adam and GD converge to the same solution with nearly identical test performance. This confirms that the generalization gap is unique to the non-convexity of deep learning landscapes.

Discussion and Limitations

While the theoretical proofs are currently limited to two-layer CNNs, the empirical results on deeper architectures like AlexNet suggest broader applicability. The results rely on a data model where noise patches have specific sparsity and correlation properties. While inspired by real-world image data, the universality of this model across other domains, such as Natural Language Processing (NLP) where Adam often excels, remains an area for future research. The authors suggest that a similar "feature learning" perspective might explain Adam's success in those domains.

Summary updated

Abstract

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed that compared with (stochastic) gradient descent, Adam can converge to a different solution with a significantly worse test error in many deep learning applications such as image classification, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the inferior generalization performance of Adam is fundamentally tied to the nonconvex landscape of deep learning optimization.
BibTeX
@inproceedings{zou2023understanding,
  author = {Difan Zou and Yuan Cao and Yuanzhi Li and Quanquan Gu},
  title = {{Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=iUYpN14qjTF},
}
Figure