Abstract
First-order optimization algorithms have been proven prominent in deep learning. In particular, algorithms such as RMSProp and Adam are extremely popular. However, recent works have pointed out the lack of “long-term memory” in Adam-like algorithms, which could hamper their performance and lead to divergence. In our study, we observe that there are benefits of weighting more of the past gradients when designing the adaptive learning rate. We therefore propose an algorithm called the Nostalgic Adam (NosAdam) with theoretically guaranteed convergence at the best known convergence rate. NosAdam can be regarded as a fix to the non-convergence issue of Adam in alternative to the recent work of [Reddi et al., 2018]. Our preliminary numerical experiments show that NosAdam is a promising alternative algorithm to Adam. The proofs, code and other supplementary materials are already released.
Introduction
Adaptive learning rate methods like Adam have become the workhorses of deep learning due to their efficiency. However, Adam relies on an Exponential Moving Average (EMA) that prioritizes recent gradients. This "short-term memory" can cause the learning rate to spike when encountering a sequence of small gradients, leading to overshooting or divergence. While AMSGrad attempted to fix this by using the maximum of past squared gradients, it can become overly restrictive—if the model hits a single massive gradient spike, the learning rate stays permanently low, slowing down training indefinitely.
The Nostalgic Approach
The core innovation of NosAdam is the "Nostalgia" weighting scheme. Instead of decaying the influence of past gradients to favor the most recent information, NosAdam ensures that the weights of past gradients are non-increasing relative to the current time step. This means the algorithm "remembers" the broader landscape it has traversed more than the immediate local fluctuations.
A specific implementation, NosAdam-HH (HyperHarmonic), uses weights following a hyper-harmonic series ($b_k = k^{-\gamma}$). This approach allows the re-scaling term to both increase and decrease, providing the flexibility to recover from gradient spikes while maintaining the stability required for a guaranteed convergence rate of $O(1/\sqrt{T})$, the best known rate for this class of stochastic optimization.
Navigating Complex Landscapes
The stability provided by long-term memory is particularly evident in complex loss landscapes. In "bowl-shaped" landscapes—where a wide global minimum is surrounded by sharp local ones—standard Adam often oscillates or gets trapped due to its sensitivity to local noise.
By weighting the past more than the present, NosAdam effectively "smooths out" these local traps. The authors demonstrate that for an adaptive optimizer to be theoretically sound, the change in the inverse learning rate must be positive semi-definite; NosAdam satisfies this naturally by prioritizing history over local dips.
Experimental Evaluation
The authors evaluated NosAdam across several benchmarks to test its robustness and performance:
- MNIST: In multi-layer networks, NosAdam demonstrated superior stability and lower final loss compared to Adam, which exhibited significant oscillation.
- CIFAR-10: Training a Wide ResNet-28, NosAdam achieved higher test accuracy (approx. 0.885) than both Adam (approx. 0.875) and AMSGrad.
- Robustness to Spikes: Qualitative experiments confirmed that NosAdam continues to converge efficiently even after encountering large gradient spikes, whereas AMSGrad's progress is often "greatly slowed down."
Conclusion and Future Work
NosAdam offers a theoretically sound and practically effective alternative to Adam. By prioritizing long-term gradient history, it achieves better stability and convergence in deep learning tasks. However, its "nostalgic" nature makes it somewhat sensitive to initialization; it requires early gradients that point in a generally helpful direction. Future work may explore the impact of the hyperparameter $\gamma$ and further decouple the "sign effect" of gradients from the adaptive rate to better understand generalization.
Summary updated