TL;DR
This paper introduces SGEM, a novel optimization algorithm that combines energy and momentum to improve the convergence and generalization of deep learning models. SGEM demonstrates unconditional energy stability and achieves faster convergence than existing methods like AEGD, while matching or exceeding the performance of SGDM on various deep learning benchmarks.
What Problem Does This Paper Address?
Training large-scale machine learning models, especially deep neural networks, often involves solving complex, non-convex optimization problems. Traditional Stochastic Gradient Descent (SGD) suffers from slow convergence due to the variance of stochastic gradients and sensitivity to learning rate tuning. While methods like momentum (e.g., SGDM) and adaptive learning rates (e.g., Adam) have been developed to address these issues, they can sometimes lead to worse generalization performance. The paper builds upon the AEGD (Adaptive Gradient Descent with Energy) method, which offers unconditional energy stability and good initial training speed, but its convergence analysis was primarily limited to deterministic settings. This work aims to combine the benefits of AEGD's energy stability with the acceleration of momentum, while also providing a theoretical framework for its performance in non-convex stochastic settings and its applicability to online convex optimization.
Methods
The core contribution is the proposed algorithm, SGEM (Stochastic Gradient with Energy and Momentum). SGEM integrates momentum into the AEGD framework by modifying the calculation of the transformed gradient ($v_t$). Specifically, it uses a momentum-based gradient estimate ($m_t$) in place of the raw stochastic gradient ($g_t$) when computing $v_t$. The update rules for the energy variable ($r_t$) and the model parameters ($\theta_t$) remain similar to AEGD.
Key aspects of the method include:
- Energy Stability: SGEM inherits the unconditional energy stability property from AEGD, meaning the energy variable $r_t$ is guaranteed to decrease, regardless of the learning rate.
- Momentum Integration: A momentum term ($m_t$) is incorporated to reduce the variance of the stochastic gradients, aiming for faster convergence.
- Unified Framework: The algorithm is analyzed for both general non-convex stochastic optimization and online convex optimization settings.
- Hyperparameter Tuning: A key advantage highlighted is that SGEM primarily requires tuning only the base learning rate ($\eta$), simplifying the hyperparameter selection process.
Datasets
The paper evaluates SGEM on several standard deep learning benchmarks:
- CIFAR-100: Used with VGG-16, ResNet-34, and DenseNet-121 architectures.
- ImageNet: Used with the ResNet-18 architecture.
The experiments are conducted with specific settings for batch size, learning rate decay, and weight decay, as detailed in the "Numerical Experiments" section.
Key Insights and Ideas
- Synergy of Energy and Momentum: The paper demonstrates that combining energy-based methods (like AEGD) with momentum can lead to a more robust and efficient optimization process, inheriting the stability of energy methods and the acceleration of momentum.
- Reduced Variance, Faster Convergence: The use of a momentum-based gradient estimate ($m_t$) significantly reduces the variance of the stochastic gradients compared to AEGD, leading to faster convergence, as evidenced by experimental results.
- Improved Generalization: Despite incorporating momentum, which can sometimes hurt generalization, SGEM is shown to generalize as well as or better than SGDM, and comparable to AEGD.
- Simplified Tuning: The claim that SGEM only requires tuning the base learning rate is a significant practical advantage, as adaptive methods often have multiple hyperparameters that are difficult to tune.
- Theoretical Guarantees: The paper provides theoretical convergence rates for SGEM in both non-convex stochastic and online convex settings, along with an unconditional energy stability guarantee.
Key Results
- Faster Convergence: SGEM converges faster than AEGD on deep learning benchmarks, as shown in Figure 1 and Figure 3.
- Competitive Test Accuracy: SGEM achieves comparable or higher test accuracy than other methods, including SGDM, on CIFAR-100 and ImageNet tasks. For example, on ImageNet with ResNet-18, SGEM achieved a test accuracy of 69.92%, very close to SGDM's 69.89%.
- Reduced Oscillation: SGEM significantly reduces the oscillation in test accuracy observed with AEGD.
- Theoretical Bounds: The paper derives energy-dependent convergence rates for the non-convex setting and a regret bound for the online convex setting.
- Energy Stability: SGEM is proven to be unconditionally energy stable.
Limitations and Future Work
- Dependence on Energy Variable ($r_t$): The current theoretical convergence bounds depend on the energy variable $r_t$. The authors suggest that deriving an asymptotic bound for $r_t$ would be beneficial for estimating adaptive learning rates.
- Energy Limiter: Since $r_t$ is strictly decreasing, there's potential to control its decay. The paper suggests that a proper energy limiter could be helpful.
- Quantifying Momentum's Effect: The paper notes that precisely quantifying the theoretical benefits of momentum on convergence bounds, even for SGDM, is an open area. SGEM's theoretical analysis could be extended to better capture this effect.
- Threshold for Learning Rate ($\eta^*$): The authors hypothesize a threshold for the learning rate $\eta^*$ where $r_t$ might decay slower than $1/\sqrt{T}$. Further investigation into this threshold and its implications is suggested.
Summary updated