Rethinking Adam: A Twofold Exponential Moving Average Approach

Yizhou Wang, Yue Kang, Can Qin, Huan Wang, Yi Xu, Yulun Zhang, Yun Fu

Abstract

Adaptive gradient methods, e.g. Adam, have achieved tremendous success in machine learning. Scaling the learning rate element-wisely by a certain form of second moment estimate of gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to suffer from compromised generalization ability compared with stochastic gradient descent (SGD) and tend to be trapped in local minima at an early stage during training. Intriguingly, we discover that substituting the gradient in the second raw moment estimate term with its momentumized version in Adam can resolve the issue. The intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a more favorable option for learning rate scaling than that of the raw gradient. Thereby we propose AdaMomentum as a new optimizer reaching the goal of training fast while generalizing much better. We further develop a theory to back up the improvement in generalization and provide convergence guarantees under both convex and nonconvex settings. Extensive experiments on a wide range of tasks and models demonstrate that AdaMomentum exhibits state-of-the-art performance and superior training stability consistently.

Introduction

In deep learning optimization, practitioners often face a dilemma: choose Stochastic Gradient Descent (SGD) for its superior generalization or Adam for its rapid convergence and ease of tuning. While Adam is the default for NLP and complex architectures, it frequently suffers from a "generalization gap" in computer vision tasks and can be prematurely trapped in poor local minima. This research identifies a fundamental flaw in Adam's design: the use of raw, noisy gradients to scale the learning rate. When gradients vanish near a basin, Adam's denominator shrinks, causing the effective step size to explode and potentially "shooting" the model out of a desirable minimum.

The AdaMomentum Optimizer

The core innovation of AdaMomentum is the Twofold Exponential Moving Average (EMA). While standard Adam maintains two moving averages—$m_t$ (the first moment) and $v_t$ (the second moment based on $g_t^2$)—AdaMomentum redefines $v_t$ as an EMA of the squared momentumized gradient ($m_t^2$).

Comparison of update rules for various optimizers, highlighting the shift from raw gradients to momentum in the second moment estimate.

Additionally, the authors propose relocating the stability constant $\epsilon$ inside the square root ($\sqrt{v_t + \epsilon}$). This minor adjustment significantly improves stability near convergence by ensuring the step size reduces more gracefully as gradients diminish.

Intuition: The "Brake" Mechanism

The momentumized gradient $m_t$ serves as a filtered, more accurate estimate of the true descent direction. In the vicinity of a minimum, the raw gradient $g_t$ vanishes quickly, but the momentum $m_t$ decays more slowly. In AdaMomentum, this persistent momentum in the denominator keeps the effective learning rate from exploding, acting as a natural "brake" that prevents overshooting.

The "brake" mechanism: AdaMomentum (blue) avoids the overshooting behavior of Adam (dark blue) by utilizing the slower decay of momentum near the global minimum.

Experimental Results

AdaMomentum was validated across a diverse set of benchmarks, including ImageNet classification (ResNet-18), NLP (Transformer-based translation), and GAN training (BigGAN).

  • Vision: On CIFAR-10 and ImageNet, AdaMomentum matched or outperformed SGDM, effectively closing the generalization gap that typically plagues adaptive optimizers in vision tasks.
  • NLP: It achieved lower perplexity on Penn Treebank and higher BLEU scores on IWSLT’14 DE-EN compared to Adam and AdamW.
  • GAN Stability: AdaMomentum demonstrated superior stability in GAN training, achieving significantly lower FID scores and more consistent convergence.
  • Landscape Analysis: 3D visualizations confirmed that AdaMomentum converges to flatter regions of the loss surface, which is a known indicator of better generalization.

Conclusion

AdaMomentum provides a robust solution to the trade-off between convergence speed and generalization. By simply momentumizing the second-moment estimate, it achieves state-of-the-art performance across vision, NLP, and generative modeling without requiring extensive hyperparameter tuning. Future work may explore its application to extreme-scale language models and integration with Nesterov acceleration.

Summary updated

Abstract

Adaptive gradient methods, e.g. \textsc{Adam}, have achieved tremendous success in machine learning. Scaling the learning rate element-wisely by a certain form of second moment estimate of gradients, such methods are able to attain rapid training of modern deep neural networks. Nevertheless, they are observed to suffer from compromised generalization ability compared with stochastic gradient descent (\textsc{SGD}) and tend to be trapped in local minima at an early stage during training. Intriguingly, we discover that substituting the gradient in the second raw moment estimate term with its momentumized version in \textsc{Adam} can resolve the issue. The intuition is that gradient with momentum contains more accurate directional information and therefore its second moment estimation is a more favorable option for learning rate scaling than that of the raw gradient. Thereby we propose \textsc{AdaMomentum} as a new optimizer reaching the goal of training fast while generalizing much better. We further develop a theory to back up the improvement in generalization and provide convergence guarantees under both convex and nonconvex settings. Extensive experiments on a wide range of tasks and models demonstrate that \textsc{AdaMomentum} exhibits state-of-the-art performance and superior training stability consistently.
BibTeX
@inproceedings{wang2022rethinking,
  author = {Yizhou Wang and Yue Kang and Can Qin and Huan Wang and Yi Xu and Yulun Zhang and Yun Fu},
  title = {{Rethinking Adam: A Twofold Exponential Moving Average Approach}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=R6hvtDTQmb},
}
Figure