Since its introduction in 2014, the Adam optimizer has become the standard for training deep learning models, from small-scale multi-layer perceptrons to massive Large Language Models (LLMs). Despite its ubiquity, a rigorous theoretical understanding of its convergence properties—particularly in comparison to the well-documented Stochastic Gradient Descent (SGD)—has remained elusive for over a decade.
In the paper "Adam symmetry theorem: characterization of the convergence of the stochastic Adam optimizer," Steffen Dereich and colleagues provide a comprehensive analysis of Adam’s convergence rates on strongly convex problems. Their work introduces a significant theoretical result: the Adam Symmetry Theorem, which suggests that Adam may not converge to the true global minimum unless the stochastic noise in the gradients is perfectly symmetrical.
The Challenge of Analyzing Adam
While the theoretical foundations of SGD are well-established, Adam is significantly more complex due to its "adaptive moment estimation." Adam maintains a moving average of both the gradients (momentum) and the squared gradients (uncentered variance). This adaptivity allows it to navigate complex loss landscapes efficiently in practice, but the non-linear nature of dividing by the square root of the second moment makes it a mathematical challenge to analyze.
Prior to this work, two major gaps existed in the literature:
- Missing Convergence Rates: There was no clear characterization of how Adam’s error scales with respect to the number of steps ($n$), the mini-batch size ($M$), and the second-moment hyperparameter ($eta_2$).
- Target Convergence: It was unclear whether Adam actually converges to the true minimizer in stochastic settings. While gradient noise in SGD typically averages out, the authors hypothesized that Adam’s update rule might "bake in" biases from the noise distribution.
Methodology: A Vector Field Approach
The authors analyze Adam through the lens of strongly convex stochastic optimization problems (SOPs). To handle the complexity of the adaptive updates, they employ a "vector field" approach, modeling the expected update step of the optimizer as a continuous-time process.
This framework allowed the derivation of explicit $L^p$ error bounds for the distance between the Adam iterates and the true global minimizer $\vartheta$. By isolating the behavior of the error as a function of the mini-batch size $M$ and the hyperparameter $eta_2$, the researchers were able to quantify exactly how these settings influence the final optimization result.
The Adam Symmetry Theorem
The most striking contribution of the paper is the Adam Symmetry Theorem. By analyzing a specific class of quadratic problems, the authors proved that the zero of the "Adam vector field"—the point toward which the optimizer naturally gravitates—only coincides with the true global minimizer if the stochastic noise is symmetrically distributed.
In most real-world datasets, gradient noise is skewed. The theorem demonstrates that for these asymmetric distributions, Adam is mathematically guaranteed to converge to a biased "wrong" point. This bias is represented in the convergence bound by an additional error term that does not exist for SGD:
$$\text{Error} \approx \frac{C_0}{M}(1 - \beta_2) + C_\beta \sqrt{\text{learning rate}}$$
This formula reveals that unless the mini-batch size $M$ approaches infinity or the hyperparameter $eta_2$ approaches 1, a persistent bias remains. This provides a theoretical justification for why batch size is uniquely critical for Adam: in SGD, larger batches primarily reduce variance, but in Adam, they are essential for reducing the inherent bias introduced by the non-linear update rule.
Key Results and Practical Implications
The study establishes several critical benchmarks for the Adam optimizer:
- Convergence Rates: The authors proved a convergence rate of $1/2$ with respect to the learning rate, matching the fundamental limit of SGD.
- Hyperparameter Scaling: They established a convergence rate of $1$ with respect to both the mini-batch size $M$ and the distance of $eta_2$ from $1$.
- Disproving Universal Convergence: The paper formally disproves the notion that Adam always converges to the true minimizer in stochastic settings, identifying the lack of symmetry as the primary driver of convergence bias.
Numerical simulations on synthetic SOPs confirmed these theoretical findings. While the bias term is often small enough to be negligible under default settings (e.g., $M=32, \beta_2=0.999$), it becomes the dominant source of error as the learning rate is decreased, potentially preventing the model from reaching the absolute optimal parameters.
Limitations and Future Directions
The primary limitation of this work is its focus on strongly convex problems. Deep learning landscapes are notoriously non-convex, and how the Adam Symmetry Theorem translates to these complex surfaces remains an open question. Future research may explore whether this observed bias acts as a form of implicit regularization that aids generalization, or if "bias-corrected" variants like AMSGrad are subject to the same symmetry constraints. For now, the theorem serves as a vital reminder that the very mechanisms that make Adam fast also introduce subtle deviations from the true mathematical optimum.
Summary updated