Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression

Xuheng Li, Quanquan Gu

Abstract

Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there have been few theoretical results explaining the effectiveness of EMA. In this paper, to better understand EMA, we establish the risk bound of online SGD with EMA for high-dimensional linear regression, one of the simplest overparameterized learning tasks that shares similarities with neural networks. Our results indicate that (i) the variance error of SGD with EMA is always smaller than that of SGD without averaging, and (ii) unlike SGD with iterate averaging from the beginning, the bias error of SGD with EMA decays exponentially in every eigen-subspace of the data covariance matrix. Additionally, we develop proof techniques applicable to the analysis of a broad class of averaging schemes.

Introduction

Exponential Moving Average (EMA) is often considered a "secret sauce" for stabilizing the training of complex architectures like GANs and diffusion models. Despite its empirical success, its theoretical foundations have remained relatively unexplored compared to other averaging techniques. Standard Stochastic Gradient Descent (SGD) without averaging converges quickly in terms of bias but suffers from high variance. Conversely, Iterate Averaging (averaging all iterates from the start) effectively reduces variance but introduces slow, polynomial bias decay, meaning the model retains influence from poor initializations for an extended period. Tail Averaging (averaging only the most recent iterates) attempts to balance these, but requires pre-specifying a training budget $N$.

This paper investigates whether EMA can provide the "best of both worlds": the rapid exponential bias convergence of standard SGD and the robust variance reduction of iterate averaging, without requiring a fixed training horizon.

Methodology

The authors analyze Online SGD with EMA within the framework of high-dimensional linear regression. This setting serves as a tractable proxy for overparameterized neural networks. The study employs a bias-variance decomposition of the excess risk and introduces a novel "triangular sum" reformulation to track error evolution across different "eigen-subspaces" of the data covariance matrix $\mathbf{H}$.

To validate the theoretical findings, synthetic experiments were conducted where data features $x_t \sim \mathcal{N}(0, \mathbf{H})$ and the covariance matrix $\mathbf{H}$ follows a decaying eigenvalue spectrum ($\lambda_i = i^{-2}$). The experiments were performed in an overparameterized regime with dimension $d = 2000$ and $N = 3000$ iterations.

Results and Analysis

The theoretical analysis reveals several key properties of EMA that distinguish it from other schemes. Most notably, EMA achieves exponential bias decay in every eigen-subspace, similar to standard SGD, while simultaneously reducing variance error.

Comparison of averaging schemes showing EMA's unique combination of exponential bias decay and reduced variance.

The "Best of Both Worlds" Property

As shown in the table above, EMA is the only scheme that maintains exponential bias decay in $N$ while achieving a variance error comparable to iterate and tail averaging. The authors identify a "phase transition" in bias decay: in directions with large eigenvalues (easy-to-learn features), bias decays at a rate governed by the EMA parameter $\alpha^{2N}$, while in directions with small eigenvalues (hard features), it follows the learning rate $(1-\delta\lambda_i)^{2N}$.

Mathematical Connections and Stability

The research establishes a mathematical bridge between EMA and tail averaging: by setting the EMA weight $\alpha$ such that $(1-\alpha)(N-s) = 1$, the variance of EMA matches that of tail averaging over the last $N-s$ iterates. However, EMA remains superior for continual learning as it does not require a fixed training horizon. Furthermore, empirical results demonstrate that EMA's bias decay is significantly smoother and more stable than that of standard SGD, which likely contributes to its success in sensitive generative modeling tasks.

Conclusion and Future Work

This work provides the first instance-dependent risk bounds for SGD with EMA in high-dimensional settings, justifying its use as a robust alternative to other averaging schemes. While these results offer strong theoretical support in the linear case, extending this analysis to non-convex landscapes typical of deep neural networks remains an open challenge. Future research may also address the small remaining gap between upper and lower bounds for specific variance terms and provide more automated ways to tune the EMA hyperparameter $\alpha$.

Summary updated

Abstract

Exponential moving average (EMA) has recently gained significant popularity in training modern deep learning models, especially diffusion-based generative models. However, there have been few theoretical results explaining the effectiveness of EMA. In this paper, to better understand EMA, we establish the risk bound of online SGD with EMA for high-dimensional linear regression, one of the simplest overparameterized learning tasks that shares similarities with neural networks. Our results indicate that (i) the variance error of SGD with EMA is always smaller than that of SGD without averaging, and (ii) unlike SGD with iterate averaging from the beginning, the bias error of SGD with EMA decays exponentially in every eigen-subspace of the data covariance matrix. Additionally, we develop proof techniques applicable to the analysis of a broad class of averaging schemes.
BibTeX
@misc{li2025understanding,
  author = {Xuheng Li and Quanquan Gu},
  title = {{Understanding SGD with Exponential Moving Average: A Case Study in Linear Regression}},
  year = {2025},
  eprint = {2502.14123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14123},
}
Figure