Abstract
Stochastic Heavy Ball (SHB) momentum is a cornerstone of modern optimization in machine learning, yet its behavior as the momentum parameter $\beta$ approaches 1 remains poorly understood. While high momentum is known to accelerate initial training, it often leads to late-stage instability and poor generalization. This paper provides the first theoretical explanation for this phenomenon, proving that high momentum amplifies stochastic noise in the final stages of optimization. To address this, we introduce a "Descending Warmup" (SHB-DW) schedule that captures early-stage acceleration without sacrificing late-stage stability.
Introduction
Training deep neural networks predominantly relies on Stochastic Gradient Descent (SGD). To speed up convergence, practitioners frequently employ Heavy Ball momentum (SHB), where a hyperparameter $\beta \in [0, 1)$ determines the influence of past gradients. While a standard value like $\beta=0.9$ is common, pushing $\beta$ closer to unity (e.g., 0.99) often results in highly oscillatory training curves and degraded final performance. This work bridges the gap between empirical observation and theoretical understanding of this "high-momentum trap."
Theoretical Analysis: The Early Bird and the Late-Stage Trap
The authors analyze SHB on quadratic, strongly convex objectives by reformulating the update rule as a matrix power series. Their findings reveal a fundamental trade-off:
- The "Early Bird" Advantage: Mathematically, as $\beta \to 1$, the iteration complexity upper bound is dominated by a smaller geometric factor than vanilla SGD. This explains why high-momentum models initially "rocket" down the loss curve.
- The Late-Stage Trap: Conversely, the authors prove a larger lower bound in complexity for SHB as $\beta \to 1$ after a certain number of iterations. High momentum effectively acts as a noise amplifier, preventing the algorithm from settling into the exact minimum and causing the observed oscillations.
Stochastic Heavy Ball with Descending Warmup (SHB-DW)
To resolve this dichotomy, the paper proposes SHB-DW. This algorithm starts with a very high momentum (e.g., $\beta=0.999$) to maximize early acceleration and then exponentially decays it to a stable baseline (e.g., $\beta=0.9$). This schedule ensures the model benefits from the "Early Bird" phase while avoiding the "Late-Stage Trap." It proves that momentum scheduling can be just as impactful as learning rate scheduling.
Empirical Results
The theory was validated on MNIST and CIFAR10 benchmarks. On CIFAR10 using ResNet18, SHB with $\beta=0.99$ was significantly outperformed by both vanilla SGD and SHB with $\beta=0.9$ due to late-stage instability.
The proposed SHB-DW algorithm achieved a top test accuracy of 93.38% on CIFAR10 (ResNet18), surpassing standard SHB (93.26%), SGD (92.66%), and Adam (91.08%). Similar gains were observed on ResNet34, demonstrating the robustness of the dynamic momentum approach.
Conclusion
This work provides a rigorous theoretical foundation for the practical limitations of high-momentum SHB. By identifying the mathematical cause of late-stage instability, the authors motivate the use of momentum scheduling. The proposed Descending Warmup strategy offers a simple yet effective way to improve both the speed and final quality of model training.
Summary updated