Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer

Steffen Dereich, Thang Do, Arnulf Jentzen, Philippe von WurstembergerarXiv 2025 · DOI · arXiv

Since its introduction in 2014, the Adam optimizer has become the standard for training deep learning models, from small-scale multi-layer perceptrons to massive Large Language Models (LLMs). Despite its ubiquity, a rigorous theoretical understanding of its convergence properties—particularly in comparison to the well-documented Stochastic Gradient Descent (SGD)—has remained elusive for over a decade.

In the paper "Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer," Steffen Dereich and colleagues provide a comprehensive analysis of Adam’s convergence rates on strongly convex problems. Their work introduces a significant theoretical result: the Adam Symmetry Theorem, which suggests that Adam may not converge to the true global minimum unless the stochastic noise in the gradients is perfectly symmetrical.

The Challenge of Analyzing Adam

While the theoretical foundations of SGD are well-established, Adam is significantly more complex due to its "adaptive moment estimation." Adam maintains a moving average of both the gradients (momentum) and the squared gradients (uncentered variance). This adaptivity allows it to navigate complex loss landscapes efficiently in practice, but the non-linear nature of dividing by the square root of the second moment makes it a mathematical challenge to analyze.

Prior to this work, two major gaps existed in the literature:

  1. Missing Convergence Rates: There was no clear characterization of how Adam’s error scales with respect to the number of steps ($n$), the mini-batch size ($M$), and the second-moment hyperparameter ($eta_2$).
  2. Target Convergence: It was unclear whether Adam actually converges to the true minimizer in stochastic settings. While gradient noise in SGD typically averages out, the authors hypothesized that Adam’s update rule might "bake in" biases from the noise distribution.

Methodology: A Vector Field Approach

The authors analyze Adam through the lens of strongly convex stochastic optimization problems (SOPs). To handle the complexity of the adaptive updates, they employ a "vector field" approach, modeling the expected update step of the optimizer as a continuous-time process.

This framework allowed the derivation of explicit $L^p$ error bounds for the distance between the Adam iterates and the true global minimizer $\vartheta$. By isolating the behavior of the error as a function of the mini-batch size $M$ and the hyperparameter $eta_2$, the researchers were able to quantify exactly how these settings influence the final optimization result.

The Adam Symmetry Theorem

The most striking contribution of the paper is the Adam Symmetry Theorem. By analyzing a specific class of quadratic problems, the authors proved that the zero of the "Adam vector field"—the point toward which the optimizer naturally gravitates—only coincides with the true global minimizer if the stochastic noise is symmetrically distributed.

In most real-world datasets, gradient noise is skewed. The theorem demonstrates that for these asymmetric distributions, Adam is mathematically guaranteed to converge to a biased "wrong" point. This bias is represented in the convergence bound by an additional error term that does not exist for SGD:

$$\text{Error} \approx \frac{C_0}{M}(1 - \beta_2) + C_\beta \sqrt{\text{learning rate}}$$

This formula reveals that unless the mini-batch size $M$ approaches infinity or the hyperparameter $eta_2$ approaches 1, a persistent bias remains. This provides a theoretical justification for why batch size is uniquely critical for Adam: in SGD, larger batches primarily reduce variance, but in Adam, they are essential for reducing the inherent bias introduced by the non-linear update rule.

Key Results and Practical Implications

The study establishes several critical benchmarks for the Adam optimizer:

  • Convergence Rates: The authors proved a convergence rate of $1/2$ with respect to the learning rate, matching the fundamental limit of SGD.
  • Hyperparameter Scaling: They established a convergence rate of $1$ with respect to both the mini-batch size $M$ and the distance of $eta_2$ from $1$.
  • Disproving Universal Convergence: The paper formally disproves the notion that Adam always converges to the true minimizer in stochastic settings, identifying the lack of symmetry as the primary driver of convergence bias.

Numerical simulations on synthetic SOPs confirmed these theoretical findings. While the bias term is often small enough to be negligible under default settings (e.g., $M=32, \beta_2=0.999$), it becomes the dominant source of error as the learning rate is decreased, potentially preventing the model from reaching the absolute optimal parameters.

Limitations and Future Directions

The primary limitation of this work is its focus on strongly convex problems. Deep learning landscapes are notoriously non-convex, and how the Adam Symmetry Theorem translates to these complex surfaces remains an open question. Future research may explore whether this observed bias acts as a form of implicit regularization that aids generalization, or if "bias-corrected" variants like AMSGrad are subject to the same symmetry constraints. For now, the theorem serves as a vital reminder that the very mechanisms that make Adam fast also introduce subtle deviations from the true mathematical optimum.

Summary updated

Abstract

Beside the standard stochastic gradient descent (SGD) method, the Adam optimizer due to Kingma & Ba (2014) is currently probably the best-known optimization method for the training of deep neural networks in artificial intelligence (AI) systems. Despite the popularity and the success of Adam it remains an \emph{open research problem} to provide a rigorous convergence analysis for Adam even for the class of strongly convex SOPs. In one of the main results of this work we establish convergence rates for Adam in terms of the number of gradient steps (convergence rate \nicefrac{1}{2} w.r.t. the size of the learning rate), the size of the mini-batches (convergence rate 1 w.r.t. the size of the mini-batches), and the size of the second moment parameter of Adam (convergence rate 1 w.r.t. the distance of the second moment parameter to 1) for the class of strongly convex SOPs. In a further main result of this work, which we refer to as \emph{Adam symmetry theorem}, we illustrate the optimality of the established convergence rates by proving for a special class of simple quadratic strongly convex SOPs that Adam converges as the number of gradient steps increases to infinity to the solution of the SOP (the unique minimizer of the strongly convex objective function) if and \emph{only} if the random variables in the SOP (the data in the SOP) are \emph{symmetrically distributed}. In particular, in the standard case where the random variables in the SOP are not symmetrically distributed we \emph{disprove} that Adam converges to the minimizer of the SOP as the number of Adam steps increases to infinity. We also complement the conclusions of our convergence analysis and the Adam symmetry theorem by several numerical simulations that indicate the sharpness of the established convergence rates and that illustrate the practical appearance of the phenomena revealed in the \emph{Adam symmetry theorem}.
BibTeX
@misc{dereich2025adam,
  author = {Steffen Dereich and Thang Do and Arnulf Jentzen and Philippe von Wurstemberger},
  title = {{Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer}},
  year = {2025},
  eprint = {2511.06675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.06675},
}
Figure