Adam Converges Without Any Modification On Update Rules

Yushun Zhang, Bingran Li, Congliang Chen, Zhi-Quan Luo, Ruoyu SunarXiv 2026 · arXiv

TL;DR

For years, a theoretical shadow loomed over the Adam optimizer due to a famous 2018 proof showing it could diverge, despite its massive practical success in training deep networks and LLMs. This paper resolves that paradox by proving that vanilla Adam is guaranteed to converge without any modifications, provided its hyperparameters are tuned after the problem and batch size are fixed. The authors identify a clear, problem-dependent "phase transition" boundary separating safe convergence from dangerous divergence in Adam's hyperparameter space.

What Problem Does This Paper Address?

In deep learning, Adam is the undisputed default optimizer, powering everything from computer vision models to massive Large Language Models (LLMs). However, an influential paper by Reddi et al. [2018] threw a wrench into its theoretical foundations by proving that Adam could diverge even on simple problems. Mysteriously, the exact hyperparameter settings flagged as unstable by Reddi et al. (like the default $\beta_1 = 0.9, \beta_2 = 0.999$) are precisely the ones that work flawlessly in practice.

This paper addresses this profound mismatch between theory and practice. The authors uncover a subtle flaw in the old divergence proof: Reddi et al. picked their adversarial problem after fixing Adam's hyperparameters $(\beta_1, \beta_2)$. In the real world, practitioners do the exact opposite—they fix the problem (and the dataset/batch size) first, and then tune the hyperparameters. This paper sets out to prove whether vanilla Adam safely converges under this realistic workflow.

Methods

The authors provide rigorous mathematical proofs of convergence for vanilla Adam under two primary data sampling strategies:

  1. With-Replacement Sampling: Where mini-batches are sampled completely at random at each step.
  2. Random Shuffling: Where the dataset is randomly permuted at the start of each epoch (the standard practice in deep learning frameworks).

The core technical challenge in proving Adam's convergence is dealing with its non-linear dynamics: the second-order momentum term ($v_k$) sits in the denominator of the update rule and can fluctuate wildly or approach zero. To overcome this without resorting to the unrealistic assumption of "bounded gradients," the authors developed a novel concentration analysis tool. They proved that when $\beta_2$ is sufficiently large, it acts as a stabilizer, slowing down changes in $v_k$ and forcing $1/\sqrt{v_k}$ to tightly concentrate around its expected value, making the optimization path predictable and stable.

Datasets

To empirically validate their theoretical findings, the authors conducted grid-search experiments using two classic machine learning benchmarks:

  • MNIST: Used to train a standard one-hidden-layer neural network.
  • CIFAR-10: Used to train a ResNet-18 architecture.

These datasets were utilized to map out the actual loss landscapes across 2,500 different combinations of $(\beta_1, \beta_2)$ to see if the empirical boundary between success and failure matched their mathematical predictions.

Key Insights and Ideas

  • The Order of Operations Matters: The fundamental reason behind the theory-practice gap is the order of picking the problem vs. the hyperparameters. Fixing the problem first guarantees that a safe, non-empty convergence region always exists.
  • The 2D Phase Transition: The paper reveals a striking "phase transition" in the $(\beta_1, \beta_2)$ plane. If $\beta_2$ is high enough and $\beta_1 < \sqrt{\beta_2}$, Adam safely converges. If $\beta_2$ drops too low, Adam enters a "danger zone" where gradients and iterates can explode to infinity.
Phase diagram showing the divergence region vs. the authors' proven convergence region, alongside empirical validation on MNIST and CIFAR-10.
  • Batch Size Dictates $\beta_2$: The boundary between the safe and dangerous zones is explicitly problem-dependent and tied to batch size. Smaller batch sizes introduce more gradient variance, which requires a larger $\beta_2$ to smooth out and ensure convergence.

Key Results

  • Guaranteed Convergence Rate: The authors prove that under proper problem-dependent tuning, Adam achieves a non-asymptotic convergence rate of $O(\frac{\log T}{\sqrt{T}})$, which matches the standard convergence rate of Stochastic Gradient Descent (SGD).
  • Neighborhood Convergence: In non-realizable settings (where data noise is present, i.e., $D_0 > 0$), Adam converges to a tight neighborhood around critical points. Crucially, the size of this neighborhood shrinks to zero as $\beta_2$ approaches 1.
  • Empirical Alignment: The grid searches on MNIST and CIFAR-10 perfectly mirror the theoretical phase diagram, showing low training loss in the predicted safe zone and high error/divergence in the small-$\beta_2$ region.
Training loss of Adam on MNIST across different batch sizes and $\beta_2$ values, showing that smaller batch sizes require larger $\beta_2$ values to converge.
  • Practical Tuning Rule: This theory yields a concrete rule of thumb for practitioners, especially when training LLMs: If your training is unstable or failing, scale up $\beta_2$ inversely with your batch size.

Limitations and Future Work

While this paper provides a massive leap forward in cementing Adam's theoretical safety, some open questions remain:

  • Tightness of Bounds: The multiplicative constants in the proven convergence upper bound depend heavily on the problem dimension ($d$) and the number of mini-batches ($n$). These bounds are likely loose and could be tightened with more fine-grained analysis.
  • Exact Boundary Geometry: The authors proved the existence of the critical boundary separating convergence and divergence, but they have not analytically mapped its exact geometric shape.
  • Adam vs. SGD: This work proves that Adam can converge as well as SGD, but it does not theoretically explain the other half of the mystery: why Adam frequently outperforms SGD in practice.

Summary updated

Abstract

Adam is the default algorithm for training neural networks, including large language models (LLMs). However, \citet{reddi2019convergence} provided an example that Adam diverges, raising concerns for its deployment in AI model training. We identify a key mismatch between the divergence example and practice: \citet{reddi2019convergence} pick the problem after picking the hyperparameters of Adam, i.e., $(β_1,β_2)$; while practical applications often fix the problem first and then tune $(β_1,β_2)$. In this work, we prove that Adam converges with proper problem-dependent hyperparameters. First, we prove that Adam converges when $β_2$ is large and $β_1 < \sqrt{β_2}$. Second, when $β_2$ is small, we point out a region of $(β_1,β_2)$ combinations where Adam can diverge to infinity. Our results indicate a phase transition for Adam from divergence to convergence when changing the $(β_1, β_2)$ combination. To our knowledge, this is the first phase transition in $(β_1,β_2)$ 2D-plane reported in the literature, providing rigorous theoretical guarantees for Adam optimizer. We further point out that the critical boundary $(β_1^*, β_2^*)$ is problem-dependent, and particularly, dependent on batch size. This provides suggestions on how to tune $β_1$ and $β_2$: when Adam does not work well, we suggest tuning up $β_2$ inversely with batch size to surpass the threshold $β_2^*$, and then trying $β_1< \sqrt{β_2}$. Our suggestions are supported by reports from several empirical studies, which observe improved LLM training performance when applying them.
BibTeX
@misc{zhang2026adam,
  author = {Yushun Zhang and Bingran Li and Congliang Chen and Zhi-Quan Luo and Ruoyu Sun},
  title = {{Adam Converges Without Any Modification On Update Rules}},
  year = {2026},
  eprint = {2603.02092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02092},
}

Loading related content...

Figure