Quasi-hyperbolic momentum and Adam for deep learning

Jerry Ma, Denis YaratsICLR 2019

In the world of deep learning, Stochastic Gradient Descent (SGD) with momentum is the reliable workhorse, while Adam is the high-performance sports car. But what if we could make both of them more robust and expressive with just a tiny change to their update rules?

In their ICLR 2019 paper, Jerry Ma and Denis Yarats introduce Quasi-Hyperbolic Momentum (QHM) and QHAdam, two optimization algorithms that provide a unifying framework for many popular optimizers while delivering state-of-the-art performance.

The Problem: The Momentum Trade-off

Standard momentum SGD (often called "Heavy Ball" momentum) uses a single parameter, $\beta$, to control two things at once: how much we "smooth" past gradients and how much the current gradient contributes to the update.

This coupling creates a fundamental trade-off. To reduce the variance of noisy gradients—a common issue in mini-batch training—you want a high $\beta$. However, a high $\beta$ makes the momentum buffer "stale," meaning it reflects the loss landscape of several steps ago rather than where the model is currently located. This staleness can lead to overshooting or training instability.

The authors aim to break this coupling, allowing researchers to benefit from the variance reduction of a long-term momentum buffer without the penalty of using stale information for the immediate update.

The Solution: Quasi-Hyperbolic Momentum

The core idea is Quasi-Hyperbolic Momentum (QHM). Instead of just using the momentum buffer to update the weights, QHM takes a weighted average of a plain SGD step and a momentum step.

The QHM Update Rule

  1. Update the momentum buffer ($g$):
    $g_{t+1} \leftarrow \beta \cdot g_t + (1 - \beta) \cdot \nabla \hat{L}_t(\theta_t)$
  2. Update the parameters ($\theta$):
    $\theta_{t+1} \leftarrow \theta_t - \alpha [(1 - \nu) \cdot \nabla \hat{L}_t(\theta_t) + \nu \cdot g_{t+1}]$

Here, $\nu$ is the immediate discount factor. By introducing $\nu$, QHM can use a very high $\beta$ (e.g., 0.999) for massive variance reduction while using a lower $\nu$ (e.g., 0.7) to ensure the current gradient still has a strong, "fresh" influence on the update.

The authors also extend this logic to Adam, creating QHAdam. It replaces both of Adam’s moment estimators (the mean and the uncentered variance) with these quasi-hyperbolic terms.

A Unifying Framework

One of the most compelling theoretical contributions of the paper is demonstrating that QHM acts as a "super-optimizer." By simply changing the values of $\nu$ and $\beta$, QHM can perfectly recover several existing optimization algorithms.

  • If $\nu = 0$, QHM becomes plain SGD.
  • If $\nu = 1$, QHM becomes standard Momentum.
  • If $\nu = \beta$, QHM recovers Nesterov Accelerated Gradient (NAG).
Table detailing how QHM relates to and recovers various optimization algorithms, including SGD, Momentum, NAG, and PID control.

This suggests that many "new" optimizers are actually just specific points within the broader QHM parameter space.

Empirical Performance and Key Results

The authors evaluated their algorithms across a diverse set of benchmarks, including image classification (EMNIST, CIFAR-10, ImageNet), language modeling (WikiText-103), reinforcement learning (MuJoCo), and neural machine translation (WMT16 EN-DE).

They provide a robust "default" setting ($\nu=0.7, \beta=0.999$) that consistently outperformed finely-tuned NAG and Adam baselines across multiple tasks.

Quantitative results comparing QHM and QHAdam against baselines across various tasks, highlighting the SOTA BLEU score and strong RL performance.

Key highlights include: * State-of-the-Art Translation: On the WMT16 EN-DE task, QHAdam achieved 29.45 BLEU, surpassing the previous state-of-the-art. * Improved Stability: While analyzing QHAdam, the authors discovered that the original Adam paper's claim about its update step size bound was incorrect. They provide a new, tighter bound and show that QHAdam’s $\nu_2$ parameter can explicitly control update stability. In the fragile Transformer training setting, QHAdam improved training stability by 40% compared to vanilla Adam. * Faster Convergence: In image recognition (ResNet-152 on ImageNet), QHM reached lower validation errors faster than the standard NAG baseline. * Reinforcement Learning Gains: In MuJoCo continuous control environments, QHAdam yielded higher average rewards in four out of seven environments tested, demonstrating its versatility beyond supervised learning.

Limitations and Future Work

While the "rule of thumb" works well in practice, QHM and QHAdam do introduce new hyperparameters ($\nu, \nu_1, \nu_2$). Finding the absolute optimal values for a novel architecture still requires a hyperparameter sweep.

Furthermore, while the authors provide strong empirical evidence and theoretical connections to other algorithms, a formal convergence analysis for QHAdam is left for future work. Finally, the paper suggests that investigating how QHM behaves in distributed, asynchronous SGD—where gradient "staleness" is an even more pronounced issue—would be a highly promising direction for future research.

Summary updated

BibTeX
@inproceedings{ma2019quasi,
  author = {Jerry Ma and Denis Yarats},
  title = {{Quasi-hyperbolic momentum and Adam for deep learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure