In deep learning, choosing an optimizer often feels like a fundamental trade-off between speed and quality. Adaptive gradient methods like Adam and RMSProp are celebrated for their rapid initial convergence, especially when navigating complex, non-convex loss landscapes. However, a persistent empirical reality complicates this success: models trained with adaptive optimizers often exhibit a significant generalization gap—achieving lower training loss but higher test error compared to those trained with simple Stochastic Gradient Descent (SGD).
Why does an algorithm that is objectively better at minimizing training loss perform worse on unseen data? The answer lies in how adaptive mechanisms alter the geometry of the loss landscape, the stochastic dynamics of training, and the implicit biases of the solutions they discover.
How Coordinate-Wise Scaling Distorts Geometry
To understand why adaptive methods struggle to generalize, we must look at how they alter the optimization path. Standard SGD updates all parameters using a uniform learning rate. In contrast, adaptive methods scale the gradient coordinate-wise, dividing the step size for each parameter by the running root-mean-square of its historical gradients.
As demonstrated by Ashia C. Wilson et al. in The Marginal Value of Adaptive Gradient Methods in Machine Learning, this coordinate-wise rescaling fundamentally shifts the optimization geometry away from maximum-margin solutions. Because adaptive methods boost updates for coordinates with small historical gradients and suppress updates for those with large gradients, they can cause the model to overfit to rare or spurious features. While this allows the optimizer to aggressively drive down training loss, it results in a solution that fails to generalize when those spurious features are absent in the test set.
The Loss of Heavy-Tailed Noise
Beyond geometry, the stochastic dynamics of the training process play a crucial role in regularization. During training, the stochasticity introduced by mini-batch sampling acts as an implicit regularizer, helping the model escape poor local minima.
Pan Zhou et al. showed in Towards Theoretically Understanding Why Sgd Generalizes Better Than Adam in Deep Learning that SGD naturally generates "heavy-tailed" gradient noise. This heavy-tailed behavior acts like a series of energetic jolts, allowing the optimizer to easily "jump out" of narrow, sharp minima and eventually settle into broad, flat basins. Flat minima are widely understood to generalize better because slight shifts between the training and testing distributions do not drastically alter the loss value.
Adaptive methods, however, use gradient smoothing and variance adaptation. As argued by the authors, this coordinate-wise scaling effectively "lightens" the tails of the gradient noise. By dampening these stochastic fluctuations, adaptive optimizers lose the ability to escape suboptimal basins, trapping the model in sharp minima that generalize poorly.
Distinct Implicit Biases and Sharpness Metrics
The choice of optimizer does not merely alter the path to a solution; it changes the destination itself. In overparameterized networks where infinite global minima exist, different optimizers exhibit distinct implicit biases.
David Reeb et al. point out in Which Minimizer Does My Neural Network Converge To? that adaptive algorithms do not merely accelerate convergence but actually settle on entirely different functional interpolators than SGD, minimizing alternative complexity measures. To pinpoint exactly what Adam minimizes, Xinghan Li et al. analyzed its behavior near the minimizer manifold in Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold. While SGD biases the model toward solutions with a low Hessian trace ($\text{tr}(\mathbf{H})$)—a classic indicator of flatness—Adam minimizes a unique sharpness measure: the trace of the square root of the Hessian's diagonal ($\text{tr}(\text{diag}(\mathbf{H})^{1/2})$). This specific bias explains why Adam excels at sparse problems (like language modeling) but falters on tasks where standard flatness is required for out-of-distribution robustness.
Furthermore, Lukas Balles et al. dissected Adam in Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients to show that its update direction is primarily determined by the sign of the gradient rather than its magnitude. While this sign-based direction accelerates early training, it is also a primary driver behind the generalization penalty.
Bridging the Gap
Recognizing these structural flaws, researchers have developed hybrid frameworks to combine the rapid convergence of adaptivity with the robust generalization of SGD. One notable breakthrough came from decoupling weight decay from the gradient updates; the authors of Decoupled Weight Decay Regularization demonstrated that standard $L_2$ regularization interacts poorly with adaptive scaling, and separating them (as in AdamW) restores significant generalization performance.
Other approaches alter the degree of adaptivity directly. For instance, the authors of Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks introduced Padam, which uses a partial adaptivity parameter to strike a balance between uniform gradient updates and coordinate-wise scaling. Similarly, Liangchen Luo et al. proposed bounding the learning rate dynamically in Adaptive Gradient Methods with Dynamic Bound of Learning Rate to transition smoothly from Adam to SGD as training progresses. These strategies underscore a growing consensus: the optimal optimization path lies somewhere between the aggressive acceleration of full adaptivity and the robust, flat-basin seeking nature of standard gradient descent.
Key Foundational Work
The Marginal Value of Adaptive Gradient Methods in Machine Learning (2017) — contributed 3 concept elements to this direction.
The Marginal Value of Adaptive Gradient Methods in Machine Learning (2017) — contributed 3 concept elements to this direction.