TL;DR
This paper introduces DEMON, a simple yet highly effective momentum decay schedule for training deep neural networks. DEMON consistently outperforms existing learning rate and momentum decay strategies across a wide range of tasks and architectures, achieving state-of-the-art results in a significant majority of experiments while also improving robustness to hyperparameter tuning.
What Problem Does This Paper Address?
Training deep neural networks (DNNs) is notoriously computationally expensive and sensitive to hyperparameter choices. While optimizers like SGD with momentum (SGDM) and Adam are widely used, achieving optimal performance often requires extensive hyperparameter tuning. Existing learning rate and momentum schedules offer various ways to adjust these parameters during training, but no single schedule is universally optimal. This paper addresses the need for a robust, easy-to-implement, and high-performing momentum decay strategy that can simplify the training process and improve generalization.
Methods
The core contribution of this paper is the DEMON (Decaying Momentum) rule. This rule proposes a simple schedule for the momentum parameter ($\beta$) that decays from an initial value (typically 0.9) to zero over the course of training. The decay is implemented as $\beta_t = \beta_{init} (1 - t/T)$, where $t$ is the current iteration and $T$ is the total number of iterations. This approach is motivated by the idea of decaying the total contribution of a gradient to future updates, aiming to stabilize weight growth and prevent suboptimal performance often seen with fixed momentum. DEMON is designed to be easily integrated into existing optimizers like SGDM and Adam with minimal computational overhead.
Datasets
The authors conducted extensive experiments across a diverse set of datasets and models, including:
- Image Classification: CIFAR-10, Tiny ImageNet, CIFAR-100, STL-10
- Language Modeling: Penn Treebank (PTB)
- Generative Models: MNIST (for VAEs and NCSN)
- Multi-task Language Understanding: GLUE benchmark
The models used spanned various architectures such as ResNets (ResNet20, ResNet56, Wide ResNet), VGG-16, LSTMs, VAEs, Capsule Networks, NCSN, and BERT.
Key Insights and Ideas
- Momentum Decay is Crucial: The paper strongly suggests that a decaying momentum schedule is not just beneficial but often superior to fixed momentum or aggressive learning rate decay alone.
- DEMON's Simplicity is its Strength: The proposed DEMON rule is remarkably simple to implement and requires no additional tuning beyond the standard optimizer parameters, making it highly practical.
- Robustness to Hyperparameter Tuning: A significant finding is DEMON's improved robustness. Experiments show that DEMON-enhanced optimizers achieve good performance across a wider range of learning rate and momentum values compared to traditional schedules, reducing the burden of extensive hyperparameter searches.
- Universal Applicability: DEMON demonstrates effectiveness across various architectures (CNNs, RNNs, Transformers) and tasks (classification, generation, language modeling), indicating its broad applicability.
Key Results
DEMON achieved remarkable success in the large-scale empirical analysis:
- Top Performance: Across 28 experimental settings, DEMON achieved the highest percentage of Top-1 finishes (39%) and Top-3 finishes (85%).
- Doubling the Competition: DEMON's Top-1 and Top-3 performance nearly doubled that of the second-best schedule, the cosine learning rate schedule (17% Top-1, 60% Top-3).
- Outperforming Baselines: DEMON consistently outperformed popular schedules like the learning rate step schedule, linear schedule, OneCycle, and exponential schedules.
- Improved Robustness: Heatmaps illustrating validation accuracy across hyperparameter grids showed that DEMON-enhanced optimizers (e.g., DEMON SGDM on WRN-STL10 and VGG16-CIFAR100) had significantly larger bands of optimal performance compared to their non-DEMON counterparts.
- Competitive Error Rates: DEMON achieved competitive generalization error rates while requiring less hyperparameter tuning.
Limitations and Future Work
- Theoretical Analysis: While the paper provides theoretical convergence analysis for DEMON with SGDM in the convex setting, the analysis for non-convex settings (especially with Adam) relies on existing work and could be further explored.
- ImageNet Performance: The authors note that for ImageNet, they did not observe significant differences between learning rate step decay and DEMON, suggesting that ImageNet results might not always generalize well.
- Future Work: The authors suggest that DEMON's effectiveness across diverse settings makes it a strong candidate for future research, particularly in exploring its integration with other advanced optimization techniques and its theoretical underpinnings in non-convex scenarios. They also highlight the potential for further research into the discrepancy observed in ImageNet performance.
Summary updated