TL;DR
This paper proposes a simple yet effective method to improve Stochastic Gradient Descent (SGD) for training deep neural networks. By aggregating model parameters at the end of distinct "phases" of training (periods of constant learning rate), the method enhances convergence and provides better initialization for subsequent phases, leading to improved performance without significant computational overhead.
What Problem Does This Paper Address?
Training deep Convolutional Neural Networks (CNNs) often relies on Stochastic Gradient Descent (SGD). While effective, standard SGD with a fixed learning rate can get stuck in "error plateaus" – suboptimal solutions. To overcome this, a common practice is to decay the learning rate over time. However, this decay is often applied uniformly, and the optimization process within each phase (defined by a constant learning rate) can still be inefficient. The paper aims to improve the overall SGD optimization by specifically addressing the behavior within these training phases, particularly around the error plateau, and leveraging the phase-wise structure of the training process.
Methods
The core of the proposed method lies in phase-wise parameter aggregation. The authors observe that within each training phase, SGD converges to an error plateau, and the optimization trajectory exhibits a certain curvature. They propose to leverage this by aggregating (averaging) the model parameters accumulated over the epochs within a phase. This aggregation is performed using an exponential moving average (EMA) at the end of each phase, formulated as:
$\theta \leftarrow \gamma \bar{\theta} + (1 - \gamma) \theta_t$
where $\theta_t$ is the current model parameter, $\bar{\theta}$ is the aggregated parameter from the previous phase, and $\gamma$ is a decaying factor. This aggregated parameter then serves as the initialization for the next training phase. The authors also note that for networks with batch normalization layers, a forward pass is performed after aggregation to recalibrate statistics. The method is designed to be simple, modifying the SGD procedure only at the end of each training phase.
Datasets
The paper evaluates its method on several standard image classification datasets:
- Cifar-100/10: 32x32 pixel images.
- SVHN: 32x32 pixel images.
- ImageNet: Larger-sized images.
Key Insights and Ideas
- Error Plateaus are Predictable: The paper provides visualizations showing that SGD, even with a decaying learning rate, converges to an error plateau within each training phase. This plateau is characterized by a specific curvature in the loss surface.
- Phase-Wise Structure is Key: Instead of viewing SGD as a monolithic process, the authors highlight the importance of its phase-wise structure (periods of constant learning rate). They argue that improving the optimization within each phase, particularly the initialization for the next, can lead to significant overall gains.
- Averaging for Better Initialization: Aggregating parameters at the end of a phase effectively creates a more favorable starting point for the subsequent phase, helping to avoid re-converging to the same error plateau or a similar suboptimal solution.
- Simplicity and Efficiency: The proposed method is remarkably simple, requiring minimal modifications to the standard SGD pipeline. It adds negligible computational overhead, as the aggregation is performed only once per phase.
Key Results
- Improved Performance: The proposed method consistently outperforms standard SGD and other related techniques (like Lookahead) across various datasets (Cifar-100/10, SVHN, ImageNet) and CNN architectures (WRN28-10, ResNet34, DenseNet121, 13-layer, ResNet50, ResNeXt50, VGG16). For example, on Cifar-100, the proposed method achieved error rates as low as 18.34% for WRN28-10, compared to 19.02% for baseline SGD.
- Robustness: The method shows robustness to different training phase durations and hyperparameters.
- Competitive with Adaptive Methods: While adaptive methods like ADAM offer faster initial convergence, the proposed method, when applied to SGD, still achieves competitive or superior final performance, especially when training for longer durations.
Limitations and Future Work
- Hyperparameter Tuning: The number of training epochs per phase ($T_p$) and the base decaying factor ($\gamma_0$) are hyperparameters that need to be tuned. While the paper suggests $\gamma_0 = 0.9$ as a good rule of thumb, optimal values might vary.
- Application to Adaptive Methods: The authors suggest that their phase-wise aggregation technique could potentially be combined with adaptive optimization methods (like ADAM) to further improve their performance, especially for longer training schedules. This is identified as a direction for future work.
Summary updated