Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate

Haiwen Huang, Chang Wang, Bin Dong

Abstract

First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of “long-term memory” in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam can be regarded as a fix to the non-convergence issue of Adam in alternative to the recent work of [Reddi et al., 2018]. Our preliminary numerical experiments show that NosAdam is a promising alternative algorithm to Adam. The proofs, code and other supplementary materials are already released.

Introduction

Adaptive learning rate methods like Adam have become the workhorses of deep learning due to their efficiency. However, Adam relies on an Exponential Moving Average (EMA) that prioritizes recent gradients. This "short-term memory" can cause the learning rate to spike when encountering a sequence of small gradients, leading to overshooting or divergence. While AMSGrad attempted to fix this by using the maximum of past squared gradients, it can become overly restrictive—if the model hits a single massive gradient spike, the learning rate stays permanently low, slowing down training indefinitely.

The Nostalgic Approach

The core innovation of NosAdam is the "Nostalgia" weighting scheme. Instead of decaying the influence of past gradients to favor the most recent information, NosAdam ensures that the weights of past gradients are non-increasing relative to the current time step. This means the algorithm "remembers" the broader landscape it has traversed more than the immediate local fluctuations.

Comparison of weighting schemes between Adam, NosAdam, and AMSGrad.

A specific implementation, NosAdam-HH (HyperHarmonic), uses weights following a hyper-harmonic series ($b_k = k^{-\gamma}$). This approach allows the re-scaling term to both increase and decrease, providing the flexibility to recover from gradient spikes while maintaining the stability required for a guaranteed convergence rate of $O(1/\sqrt{T})$, the best known rate for this class of stochastic optimization.

The stability provided by long-term memory is particularly evident in complex loss landscapes. In "bowl-shaped" landscapes—where a wide global minimum is surrounded by sharp local ones—standard Adam often oscillates or gets trapped due to its sensitivity to local noise.

Optimization trajectory on a bowl-shaped landscape.

By weighting the past more than the present, NosAdam effectively "smooths out" these local traps. The authors demonstrate that for an adaptive optimizer to be theoretically sound, the change in the inverse learning rate must be positive semi-definite; NosAdam satisfies this naturally by prioritizing history over local dips.

Experimental Evaluation

The authors evaluated NosAdam across several benchmarks to test its robustness and performance:

  • MNIST: In multi-layer networks, NosAdam demonstrated superior stability and lower final loss compared to Adam, which exhibited significant oscillation.
  • CIFAR-10: Training a Wide ResNet-28, NosAdam achieved higher test accuracy (approx. 0.885) than both Adam (approx. 0.875) and AMSGrad.
  • Robustness to Spikes: Qualitative experiments confirmed that NosAdam continues to converge efficiently even after encountering large gradient spikes, whereas AMSGrad's progress is often "greatly slowed down."

Conclusion and Future Work

NosAdam offers a theoretically sound and practically effective alternative to Adam. By prioritizing long-term gradient history, it achieves better stability and convergence in deep learning tasks. However, its "nostalgic" nature makes it somewhat sensitive to initialization; it requires early gradients that point in a generally helpful direction. Future work may explore the impact of the hyperparameter $\gamma$ and further decouple the "sign effect" of gradients from the adaptive rate to better understand generalization.

Summary updated

Abstract

First-order optimization algorithms have been proven prominent in deep learning. In particu- lar, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of “long-term memory” in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam can be regarded as a fix to the non-convergence issue of Adam in alternative to the recent work of [Reddi et al., 2018]. Our preliminary numerical experiments show that NosAdam is a promising alternative al- gorithm to Adam. The proofs, code and other supplementary materials are already released.
BibTeX
@inproceedings{huang2019nostalgic,
  author = {Haiwen Huang and Chang Wang and Bin Dong},
  title = {{Nostalgic Adam: Weighting More of the Past Gradients When Designing the Adaptive Learning Rate}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=J7pi3-JGEAD},
}
Figure