TL;DR
This paper challenges the common belief that momentum in stochastic gradient descent (SGD) reduces stochastic noise and improves generalization. Through theoretical analysis and experiments, the authors demonstrate that momentum does not significantly reduce gradient noise or search direction noise. Instead, they propose that the benefits of momentum, if any, are limited and primarily related to the degree of smoothing introduced by search direction noise, which is not enhanced by momentum itself.
What Problem Does This Paper Address?
For years, a widely held assumption in deep learning optimization has been that adding momentum to stochastic gradient descent (SGD) helps to "smooth out" the noisy gradients encountered during training, leading to faster convergence and better generalization. This belief is particularly prevalent when training complex, non-convex objective functions like those found in deep neural networks. However, the theoretical underpinnings of this phenomenon have been less clear, and experimental results have sometimes seemed contradictory. This paper aims to rigorously investigate the role of momentum in reducing stochastic noise and its impact on generalization, specifically questioning whether the common wisdom holds true.
Methods
The authors employ a multi-pronged approach:
- Convergence Analysis: They analyze the convergence properties of SGD with Normalized-Stochastic Heavy Ball (NSHB) momentum, a common implementation of momentum.
- Optimal Batch Size Estimation: They derive a novel formula for estimating the optimal batch size, which is crucial for understanding the interplay between batch size, learning rate, and noise.
- Gradient Noise Analysis: They define and quantify "gradient noise" as the difference between the mini-batch gradient and the true gradient. They then analyze how momentum affects this noise.
- Search Direction Noise Analysis: They introduce and analyze "search direction noise," a different formulation of stochastic noise representing the deviation from the steepest descent direction. They show this noise inherently smooths the objective function.
- Degree of Smoothing: They derive a measure for the "degree of smoothing" introduced by search direction noise and investigate how momentum affects it.
- Experimental Validation: They conduct extensive experiments using ResNet18, WideResNet-28-10, and MobileNetV2 on CIFAR100 and CIFAR10 datasets to validate their theoretical findings.
Datasets
- CIFAR100: Used for training and evaluating ResNet18, WideResNet-28-10, and MobileNetV2.
- CIFAR10: Used for training and evaluating ResNet18.
Key Insights and Ideas
- Momentum Doesn't Reduce Gradient Noise: Contrary to popular belief, the paper provides theoretical and experimental evidence that momentum does not significantly reduce the magnitude of gradient noise. The noise levels for SGD and NSHB are found to be comparable.
- Search Direction Noise Smooths Objectives: The authors introduce "search direction noise" and show that it inherently smooths the objective function. This smoothing is proposed as a mechanism for improved generalization.
- Momentum Doesn't Enhance Smoothing: Crucially, the degree of smoothing introduced by search direction noise is found to be independent of the momentum factor. This means momentum doesn't contribute to this beneficial smoothing effect.
- Generalization is Linked to Smoothing, Not Momentum: The paper suggests that generalization performance is more strongly correlated with the degree of smoothing achieved by search direction noise than with the presence of momentum itself.
- Learning Rate is Key: The effectiveness of optimizers, including momentum-based ones, is highly dependent on the learning rate. When the learning rate is not appropriate, even with momentum, the expected benefits might not materialize.
Key Results
- Gradient Noise: Experiments showed that SGD and NSHB (SGD with momentum) produced similar levels of gradient noise across various learning rates and batch sizes.
- Search Direction Noise: The magnitude of search direction noise for NSHB was found to have the same upper bound as for SGD, indicating no reduction due to momentum.
- Degree of Smoothing: The degree of smoothing was shown to be dependent on the learning rate and batch size, but not on the momentum factor.
- Test Accuracy vs. Smoothing: Test accuracy was found to be a concave function of the degree of smoothing, suggesting an optimal range for smoothing.
- SGD vs. NSHB Performance: For small learning rates, SGD and NSHB showed similar test accuracies. However, for large learning rates, NSHB sometimes outperformed SGD, but this was attributed to the learning rate's effectiveness rather than momentum's noise-reducing properties.
Limitations and Future Work
- Learning Rate Dependency: The paper acknowledges a limitation in its theory: the explanation of test accuracy by the degree of smoothing breaks down when the learning rate is not effective for the optimization process. This means that while a certain degree of smoothing might be optimal, achieving it with an inappropriate learning rate doesn't guarantee good performance.
- Hyperparameter Tuning: Deriving the optimal degree of smoothing without prior calculation is currently impossible, representing an important area for future work. This could save significant computational resources in hyperparameter tuning.
- "Appropriate" Learning Rates: The paper highlights that for large learning rates, NSHB can outperform SGD, but this behavior is not fully explained by the current theory and suggests that the definition of an "effective" learning rate is critical.
- Common Belief Disproven: The authors explicitly state that their results disprove the common belief that momentum reduces stochastic noise. This is a significant finding that may prompt a re-evaluation of how momentum is understood and applied.
Summary updated