Central limit theorem for the averaged Adam optimizer

Steffen Dereich, Arnulf JentzenarXiv 2026 · arXiv

TL;DR

This paper provides the first mathematical proof of a Central Limit Theorem (CLT) for the averaged Adam optimizer. It demonstrates that applying Ruppert-Polyak averaging to Adam achieves the optimal $n^{-1/2}$ convergence rate, matching classical stochastic gradient descent (SGD) benchmarks.

What Problem Does This Paper Address?

Adam is the go-to optimizer for training modern artificial intelligence systems, yet its theoretical properties remain notoriously difficult to analyze compared to standard SGD. One major gap is understanding its asymptotic behavior when its iterations are averaged—a technique known as Ruppert-Polyak averaging, which famously optimizes SGD's convergence speed to $n^{-1/2}$.

Because Adam adjusts its learning rates dynamically using a running history of past gradients, it possesses a "memory" that turns it into a complex, fast-slow dynamical system. Previous studies could only provide loose upper bounds on Adam's error under restrictive conditions (such as shifting optimizer parameters over time). This paper tackles the challenge of proving a rigorous Central Limit Theorem for averaged Adam to pin down its exact speed of convergence and asymptotic distribution.

Methods

The authors use a purely theoretical probabilistic framework to analyze the algorithm. The core innovation is the introduction of a martingale correction tool designed specifically to handle the memory delays inherent in Adam's architecture.

The mathematical approach consists of:

  1. Linearizing the non-linear Adam vector field around its limit point (a stable attractor) using a first-order Taylor expansion.
  2. Introducing a martingale correction term to perform a perturbation analysis of the state parameters.
  3. Applying a martingale-coboundary decomposition to break down the complex, history-dependent errors into a stationary martingale difference sequence. This allows the authors to deploy a classical martingale Central Limit Theorem.

Datasets

Not discussed in the paper. This is a purely theoretical mathematics and probability paper, so no empirical datasets or benchmarks were used.

Key Insights and Ideas

  • Restoring Optimality: Standard Adam typically exhibits a sub-optimal convergence rate of $n^{-\nu/2}$ (where $\nu < 1$). This paper proves that averaging the iterates successfully bumps the convergence speed up to the optimal $n^{-1/2}$ rate seen in classical stochastic approximation.
  • The Cost of Memory: Unlike standard SGD, where errors can be easily treated as martingale differences, Adam's history-dependent scaling requires a brand-new mathematical tool—the martingale correction—to smooth out the perturbations caused by past states.
  • Attractor vs. Minimum: The paper highlights that Adam converges to a zero of the Adam vector field (an attracting state), which is distinct from a simple critical point of the objective function itself, underscoring the unique implicit biases of adaptive optimizers.

Key Results

The main result (Theorem 2.6) proves that the $\sqrt{n}$-scaled error of the averaged Adam optimizer converges weakly to a centered normal distribution as the number of steps $n$ goes to infinity:

$$\sqrt{n}(\bar{\theta}_n - \theta^*) \Rightarrow \mathcal{N}(0, H^{-1}\Gamma(H^{-1})^\dagger)$$

Where:

  • $\bar{\theta}_n$ is the averaged Adam trajectory.
  • $\theta^*$ is the limit point (stable attractor) of the Adam vector field.
  • $H$ represents the matrix from the linearization of the vector field at the attractor.
  • $\Gamma$ is the explicitly specified covariance matrix determined by the algorithm's properties at that attractor.

This formula mathematically guarantees an exact convergence speed of $n^{-1/2}$.

Limitations and Future Work

While this work establishes a foundational theoretical baseline, it focuses on an equally weighted average of the entire history of Adam steps. The authors openly acknowledge that in practical deep learning scenarios (like training deep neural networks), equally weighted averages can actually slow down optimization because the trajectories often traverse highly non-convex landscapes or diverge.

Real-world applications typically favor Exponential Moving Averages (EMAs) or tail-averaging (e.g., averaging only the last 1000 steps). Extending this rigorous CLT framework to EMAs, schedule-free variants, and non-convex landscapes remains an open and vital direction for future research.

Summary updated

Abstract

In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quantifies exactly the speed of convergence. The order of convergence is $n^{-1/2}$ in the number of steps of the algorithm which coincides with the order observed for classical stochastic approximation algorithms. The covariance in the central limit theorem is given in terms of properties of the Adam algorithm in the state of the attractor.
BibTeX
@misc{dereich2026central,
  author = {Steffen Dereich and Arnulf Jentzen},
  title = {{Central limit theorem for the averaged Adam optimizer}},
  year = {2026},
  eprint = {2606.21433},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.21433},
}
Figure