Exploring the Inefficiency of Heavy Ball as Momentum Parameter Approaches 1

Xiaoge Deng, Tao Sun, Dongsheng Li, Xicheng Lu

Abstract

The heavy ball momentum method is a commonly used technique for accelerating training processes in the machine learning community. However, empirical evidence suggests that the convergence of stochastic gradient descent (SGD) with heavy ball may slow down when the momentum hyperparameter approaches 1. Despite this observation, there are no established theories or solutions to explain and address this issue. In this study, we provide the first theoretical result that elucidates why momentum slows down SGD as it tends to 1. To better understand this inefficiency, we focus on the quadratic convex objective in the analysis. Our findings show that momentum accelerates SGD when the scaling parameter is not very close to 1. Conversely, when the scaling parameter approaches 1, momentum impairs SGD and degrades its stability. Based on the theoretical findings, we propose a descending warmup technique for the heavy ball momentum, which exploits the advantages of the heavy ball method and overcomes the inefficiency problem when the momentum tends to 1. Numerical results demonstrate the effectiveness of the proposed SHB-DW algorithm.

Introduction: The Momentum Paradox

In modern deep learning, "vanilla" Stochastic Gradient Descent (SGD) is rarely used in isolation. Instead, practitioners rely on momentum-based variants like the Stochastic Heavy Ball (SHB) method to navigate complex loss landscapes. SHB utilizes a hyperparameter $\beta$ (typically 0.9) to incorporate previous update directions, which helps the optimizer traverse flat regions and accelerate convergence.

However, a persistent empirical headache has been the "momentum paradox": as $\beta$ is increased toward 1 (e.g., 0.99 or 0.999) to gain more acceleration, the training process often becomes highly oscillatory, slows down significantly, or fails to generalize. This paper provides the first rigorous theoretical framework to explain this inefficiency and offers a practical algorithmic solution.

Theoretical Framework: Analyzing the Quadratic Objective

To understand the dynamics of SHB, the authors analyze it through the lens of a quadratic convex objective function. This simplification allows the SHB update rule to be modeled as a linear dynamical system. By examining the spectral properties of the transition matrix governing error evolution, the researchers derived key upper and lower bounds for convergence.

Key Insights: The Early Advantage vs. The Late-Stage Floor

The analysis reveals that momentum's utility is highly time-dependent. When the initial error is large, a $\beta$ close to 1 significantly reduces the "geometric factor" of the convergence rate, explaining why high momentum feels fast at the start of training.

Convergence behavior across different $\beta$ values, illustrating the trade-off between early speed and late-stage precision.

However, Theorem 2 in the paper identifies a "late-stage floor." As $\beta \to 1$, the lower bound of the error becomes dominated by a term related to the noise variance. This means that SHB with extreme momentum will never reach the same precision as vanilla SGD, regardless of the number of iterations. As seen in the figure above, while higher momentum (e.g., $\beta=0.9$) may drop faster initially, it eventually plateaus at a higher error level than more moderate momentum values.

Instability and Performance Degradation

Beyond slowing down, extreme momentum introduces significant instability. The paper mathematically demonstrates that when $\beta$ is too high, the system's "memory" becomes too heavy, causing the optimizer to repeatedly overshoot the minimum.

Test accuracy curves for different momentum parameters $\beta$. Note the high oscillations and lower final performance of $\beta=0.99$ compared to $\beta=0.9$.

This instability is clearly visible in empirical tests on benchmarks like MNIST and CIFAR-10. As shown in the test accuracy curves above, the $\beta=0.99$ configuration (cyan line) exhibits massive oscillations and fails to reach the peak accuracy achieved by the standard $\beta=0.9$ setting.

Proposed Solution: Descending Warmup (SHB-DW)

Based on the finding that high momentum is beneficial only in the early stages, the authors propose Stochastic Heavy Ball with Descending Warmup (SHB-DW). Unlike standard warmup techniques that focus on the learning rate, SHB-DW starts with a very high initial momentum (e.g., $\beta=0.999$) to exploit early-stage acceleration and then decays it according to a descending factor until it reaches a stable, lower value (e.g., $\beta=0.9$).

Key Results

  • Superior Generalization: On ResNet18 (CIFAR-10), SHB-DW achieved a test accuracy of 93.38%, outperforming both vanilla SGD (92.66%) and standard SHB with $\beta=0.9$ (93.26%).
  • Stability: SHB-DW remained as stable as vanilla SGD while maintaining the speed of high-momentum methods, effectively acting as a "booster rocket" that detaches to allow for a smooth landing.

Conclusion

This work clarifies the theoretical limits of the Heavy Ball method, showing that extreme momentum is a double-edged sword that provides early speed at the cost of late-stage stability and precision. The proposed SHB-DW algorithm offers a simple yet effective way to harness the best of both worlds. Future work may extend this analysis to Nesterov’s Accelerated Gradient (NAG) and explore combinations with adaptive learning rate methods like Adam.

Summary updated

Abstract

The heavy ball momentum method is a commonly used technique for accelerating training processes in the machine learning community. However, empirical evidence suggests that the convergence of stochastic gradient descent (SGD) with heavy ball may slow down when the momentum hyperparameter approaches 1. Despite this observation, there are no established theories or solutions to explain and address this issue. In this study, we provide the first theoretical result that elucidates why momentum slows down SGD as it tends to 1. To better understand this inefficiency, we focus on the quadratic convex objective in the analysis. Our findings show that momentum accelerates SGD when the scaling parameter is not very close to 1. Conversely, when the scaling parameter approaches 1, momentum impairs SGD and degrades its stability. Based on the theoretical findings, we propose a descending warmup technique for the heavy ball momentum, which exploits the advantages of the heavy ball method and overcomes the inefficiency problem when the momentum tends to 1. Numerical results demonstrate the effectiveness of the proposed SHB-DW algorithm.
BibTeX
@inproceedings{deng2024exploring,
  author = {Xiaoge Deng and Tao Sun and Dongsheng Li and Xicheng Lu},
  title = {{Exploring the Inefficiency of Heavy Ball as Momentum Parameter Approaches 1}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=KWs7uv5QlA},
}
Figure