Closing the Generalization Gap in Adaptive Optimization

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2287 papers386 conceptsfrozen 2026-10-04

In the toolkit of modern deep learning, practitioners often face a binary choice: the rapid, "set-it-and-forget-it" convergence of adaptive optimizers like Adam, or the superior final accuracy of carefully tuned Stochastic Gradient Descent (SGD). While Adam is the "Swiss Army knife" of the field, it consistently fails to reach the same level of out-of-sample generalization as SGD, particularly in computer vision and complex transformer tasks.

Recent research has moved beyond simply observing this "generalization gap" to dissecting its mechanical and geometric causes, proposing new algorithms that aim to combine adaptive speed with SGD-like robustness.

The Anatomy of the Gap: Sign and Variance

To understand why adaptive methods struggle, we must look at what they actually do to the gradient. Lukas Balles and Philipp Hennig in Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients reveal that Adam’s update is composed of two distinct parts: a direction based on the sign of the gradient and a magnitude based on variance adaptation.

While variance adaptation helps navigate poorly scaled dimensions, the reliance on the gradient's sign is a double-edged sword. Sign-based updates allow for rapid initial progress, but they are the primary culprit behind poor generalization. Theoretical work by Bingrui Li and colleagues in On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent suggests that sign-based methods tend to "memorize" noise patterns in the training data rather than learning the underlying signal, leading to near-zero training loss but high test error.

The Geometry of Sharpness

A recurring theme across the literature is the "sharpness" of the minima found by different optimizers. It is a well-established heuristic that "flat" minima—regions where the loss doesn't change much if you move the weights slightly—generalize better than "sharp" minima.

Adaptive optimizers seem drawn to sharpness. Jeremy Cohen and Naman Agarwal show in Adaptive Gradient Methods at the Edge of Stability that adaptive methods operate at the "Adaptive Edge of Stability." Unlike SGD, which is repelled by high-curvature regions, adaptive methods "cheat" by constantly reshaping the optimization landscape. Their per-parameter scaling allows them to tolerate higher curvature, effectively allowing them to dive into sharp crevices of the loss surface that SGD would simply bounce out of.

This is further supported by Abulikemu Abuduweili and Changliu Liu, who found that Adam’s initial steps—often starting from a zero-initialized second moment—trap the model in less robust regions of the landscape early on Revisiting the Initial Steps in Adaptive Gradient Descent Optimization.

Bridging the Gap: Partial Adaptivity and Hybrid Methods

If the problem is "too much adaptivity," the solution might be to dial it back. Several researchers have proposed "hybrid" optimizers that sit on a spectrum between Adam and SGD:

Practical Workflows: The "Switch" Strategy

While new algorithms emerge, a common practical consensus has formed: start with adaptivity, end with momentum. As noted in Introduction to deep learning by Lihi Shiloh-Perl and Raja Giryes, many state-of-the-art results are achieved by beginning training with Adam to navigate the initial high-loss plateau quickly, then switching to SGD with momentum to "settle" into a high-quality, flat minimum for final generalization.

This evolution in understanding suggests that the "gap" is not an inherent flaw of adaptivity itself, but a consequence of how coordinate-wise scaling interacts with the geometry of neural loss landscapes. By calibrating these updates—whether through bounding learning rates (AdaMod) or centering momentum (ACProp)—the field is moving toward a single, robust optimization strategy that requires no compromise.

Papers in this version

Showing 60 of 287 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 203 papers
  2. Version 1 · 2026-10-04 · 287 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_adaptive_optimizer_generalization_gap_987823c1,
  title = {adaptive optimizer generalization gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-optimizer-generalization-gap/rdr_987823c147d94e24811a97ef92c655fd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_closing_the_generalization_gap_in_adapti_987823c1_v1,
  title = {Closing the Generalization Gap in Adaptive Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/closing-the-generalization-gap-in-adaptive-optimization/rdr_987823c147d94e24811a97ef92c655fd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}