Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization

Difan Zou, Yuan Cao, Yuanzhi Li, Quanquan Gu

In the world of deep learning, choosing an optimizer is often a trade-off between speed and quality. Adam is the go-to algorithm for fast convergence, but practitioners frequently switch to Stochastic Gradient Descent (SGD) for the final stages of training. Why? Because SGD consistently finds solutions that generalize better to unseen data.

The ICLR 2023 paper, "Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization," provides a rigorous theoretical explanation for this phenomenon. It reveals that Adam’s "coordinate-wise" nature makes it a magnet for noise in the complex, nonconvex landscapes of deep neural networks.

The Generalization Gap

For years, researchers have observed a persistent "generalization gap": models trained with adaptive methods like Adam often yield higher test errors than those trained with SGD, even when both achieve zero training error.

Test accuracy of SGD vs. Adam on CIFAR-10 across different architectures.

Previous theories suggested this gap was due to "implicit bias"—an algorithm's inherent preference for certain types of solutions. However, these theories primarily examined simple linear models or convex problems. In those simpler cases, adding weight decay (a standard regularization technique) forces all optimizers to converge to the same unique solution, effectively erasing the gap.

Yet, in real-world deep learning, the gap persists even with optimal weight decay. This paper sets out to explain why.

The Theoretical Framework: Features vs. Noise

To understand this divergence, the authors move beyond linear models and analyze a two-layer Convolutional Neural Network (CNN) trained on a specially designed "image-like" dataset.

In their theoretical model, each data point consists of two distinct components:

  1. A Feature Patch: A clear, concentrated signal (akin to a distinct shape or edge) that is highly correlated with the correct label.
  2. A Noise Patch: A collection of random, high-dimensional, diffuse "noise" coordinates.

The researchers mathematically track how Adam and Gradient Descent (GD) behave during two phases of training: the initial pattern learning phase, and the subsequent regularization phase where weight decay pulls weights toward zero.

The "Coordinate-Wise" Trap

The core insight of the paper lies in how each optimizer processes gradients:

  • Adam's Noise Sensitivity: Adam updates every weight by normalizing it against its historical gradient. Because noise in images is typically spread across many coordinates ("dense noise"), Adam amplifies these small, diffuse signals and attempts to memorize them.
  • GD’s Signal Priority: Gradient Descent, on the other hand, relies on the $L_2$ norm (the total magnitude) of the gradient. Because the feature patch provides a concentrated, strong signal, GD is naturally drawn to the structural features and largely ignores the diffuse background noise.
First-layer filters learned by Adam, exhibiting noisy, speckled patterns.

This divergence leads to a phenomenon the authors call "flipped feature learning," where Adam actually learns to associate the noise with the label more strongly than the true underlying features. Visualizations of the first-layer filters learned by Adam on CIFAR-10 confirm this, appearing as noisy speckles compared to the smoother, structured patterns learned by SGD.

The Role of Nonconvexity

Crucially, the authors prove that the optimization landscape itself is the deciding factor:

  • In a Convex Landscape (Theorem 4.2): There is only one global minimum. Weight decay makes this "valley" steeper, forcing both Adam and GD to converge to the exact same solution with the same test error.
  • In a Nonconvex Landscape (Theorem 4.1): Deep learning models have many valleys that all result in zero training error. Because Adam and GD process signals differently, their algorithmic biases steer them toward entirely different valleys. The authors mathematically prove that in their CNN setting, GD achieves nearly zero test error, while Adam’s test error remains no better than a random guess ($\ge 0.5$), even with optimal weight decay.

Limitations and Future Directions

While the paper provides a robust mathematical foundation for the generalization gap, the primary proofs are limited to two-layer networks. Extending these proofs to very deep architectures like ResNets remains an open challenge, though empirical simulations support the theory.

Furthermore, the theory relies on features being sparse and noise being dense—a common characteristic of image data. In fields like Natural Language Processing (NLP), where Adam frequently outperforms SGD, the data structure is fundamentally different. Investigating how data sparsity interacts with adaptive optimizers could pave the way for new algorithms that combine Adam's speed with SGD's structural prioritization.

Summary updated

Abstract

Adaptive gradient methods such as Adam have gained increasing popularity in deep learning optimization. However, it has been observed in many deep learning applications such as image classification, Adam can converge to a different solution with a worse test error compared to (stochastic) gradient descent, even with a fine-tuned regularization. In this paper, we provide a theoretical explanation for this phenomenon: we show that in the nonconvex setting of learning over-parameterized two-layer convolutional neural networks starting from the same random initialization, for a class of data distributions (inspired from image data), Adam and gradient descent (GD) can converge to different global solutions of the training objective with provably different generalization errors, even with weight decay regularization. In contrast, we show that if the training objective is convex, and the weight decay regularization is employed, any optimization algorithms including Adam and GD will converge to the same solution if the training is successful. This suggests that the generalization gap between Adam and SGD in the presence of weight decay regularization is closely tied to the nonconvex landscape of deep learning optimization, which cannot be covered by the recent neural tangent kernel (NTK) based analysis.
BibTeX
@inproceedings{zou2023understanding,
  author = {Difan Zou and Yuan Cao and Yuanzhi Li and Quanquan Gu},
  title = {{Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=iUYpN14qjTF},
}
Figure