TL;DR
Adaptive gradient optimizers like Adam converge rapidly but often generalize worse than SGD, partly because raw gradient variance distorts their stepsize preconditioner. AdaMomentum resolves this by replacing the raw gradient in the second-moment estimate ($v_t$) with its momentumized counterpart ($m_t$), smoothing the denominator and dynamically scaling stepsizes to better navigate loss landscapes. This simple drop-in modification matches or beats SGD on vision benchmarks while preserving the rapid convergence of adaptive methods in NLP and generative modeling.
What Problem Does This Paper Address?
Deep learning practitioners face a classic optimizer dilemma:
- Stochastic Gradient Descent with Momentum (SGDM) typically generalizes better and finds flatter minima in computer vision, but it adapts slowly to ill-conditioned loss curvatures.
- Adaptive methods (e.g., Adam) train significantly faster on complex architectures (Transformers, RNNs, GANs) by scaling updates elementwise using past squared gradients, but they frequently suffer from a generalization gap and can get trapped in suboptimal local minima early on.
Previous fixes (such as switching from Adam to SGD or adding dynamic clipping bounds) often require delicate hyperparameter tuning or only work well on narrow subsets of tasks. The authors ask: Can a computationally lightweight change to Adam achieve fast optimization and superior generalization without tuning headaches?
Methods
AdaMomentum makes two targeted modifications to standard Adam:
-
Second-Moment Scaling via Momentumized Gradients:
Standard Adam tracks the exponentially decaying average of raw squared gradients: $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2$$ AdaMomentum instead feeds the first-moment estimate ($m_t$) into the second-moment accumulator: $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) m_t^2 + \epsilon$$ Because $m_t$ already filters out high-frequency gradient noise, its running squared magnitude yields a more stable, directionally accurate preconditioner. -
Damping Term Placement:
The damping parameter $\epsilon$ is shifted from outside the square root in the update denominator ($\sqrt{\hat{v}_t} + \epsilon$) to inside the recursive formula for $v_t$. Near convergence, where gradient magnitudes vanish, this small shift effectively reduces the step size by an extra factor of $\approx \sqrt{1 - \beta_2}$, dampening late-stage parameter oscillations.
The authors prove convergence guarantees for AdaMomentum:
- Convex Regret: Bounds regret to $\tilde{\mathcal{O}}(\sqrt{T})$.
- Non-Convex Convergence: Achieves an $\mathcal{O}(\log(T)/\sqrt{T})$ rate for $\min_{t \in [T]} \mathbb{E}[\|\nabla f(\theta_t)\|_2^2]$.
- Generalization Mechanics: Modeling parameter trajectories with heavy-tailed Lévy-driven stochastic differential equations (SDEs), they prove that AdaMomentum has a longer expected escaping time than Adam in flat or asymmetric basins—the types of minima associated with superior test performance.
Datasets
The authors evaluated AdaMomentum across diverse domains without introducing new datasets:
- Synthetic 2D Landscapes: Sphere, Three-Hump Camel, Beale, and Ackley functions.
- Computer Vision: CIFAR-10 (Spectral Normalized GAN, VGGNet-16, ResNet-34, DenseNet-121) and ImageNet (ResNet-18).
- Language Modeling: Penn Treebank (1-, 2-, and 3-layer LSTMs).
- Machine Translation: IWSLT’14 German-to-English (Transformer Small).
Key Insights and Ideas
- Better Curvature Responsiveness: When exiting a flat region into a steep cliff, raw $|g_t|$ spikes immediately, blowing up Adam's denominator and artificially suppressing the step size. Because $m_t$ lags behind the raw spike via exponential smoothing, AdaMomentum keeps the step size larger to traverse the slope quickly. Conversely, when approaching a basin, $m_t$ remains larger than the suddenly diminished $g_t$, keeping step sizes cautious and preventing the optimizer from overshooting the valley.
- Noise Filtration in the Preconditioner: High mini-batch gradient variance corrupts $v_t$ in Adam. Using $m_t^2$ naturally acts as a low-pass filter, dampening orthogonal stochastic noise.
- Zero Additional Memory or Compute Overhead: Because $m_t$ is already computed for the numerator, reusing it in the denominator incurs virtually zero computational overhead and requires no extra state tensors.
Key Results
-
Computer Vision (Closing the SGD Gap):
Adaptive methods historically trail SGDM on CIFAR-10 and ImageNet. AdaMomentum matches or slightly edges out SGDM: -
ImageNet (ResNet-18): AdaMomentum hits 70.45% top-1 accuracy, outperforming SGDM (70.41%), AdamW (68.77%), and AdaBelief (69.98%).
- CIFAR-10 (ResNet-34 / DenseNet-121): Scores 96.33% and 95.08%, compared to SGDM's 96.47% and 95.03%, while Adam reaches only 95.39% and 93.92%.
-
Language Modeling (Penn Treebank):
AdaMomentum achieves the lowest test perplexities across all model depths (1-layer: 80.82 vs. Adam's 84.55; 3-layer: 60.08 vs. Adam's 64.10). -
GAN Stability:
On CIFAR-10 with Spectral Normalized GANs, AdaMomentum reaches a Frechet Inception Distance (FID) of 12.06 ± 0.21, outperforming AdamW (13.05) and AdaBelief (12.52). -
Tuning-Friendly: Standard default hyperparameters ($\alpha=0.001, \beta_1=0.9, \beta_2=0.999$) worked consistently well across vision and NLP tasks without requiring aggressive per-task tuning.
Limitations and Future Work
- Exploration of Nesterov Momentum: The paper focuses on classical Polyak-style momentum; integrating Nesterov-accelerated momentum (similar to NAdam) remains unexplored.
- Large-Scale Multi-Modal and LLM Pretraining: Evaluations covered moderate-sized Transformer and CNN models (e.g., ResNet-18/34, small Transformer on IWSLT'14). Validating scalability on modern multi-billion-parameter language models and large batch training setups is left for future research.
Summary updated