TL;DR
This paper introduces Stochastic Weight Averaging (SWA), a simple yet effective technique that averages model weights from different points during SGD training. SWA consistently improves test accuracy and generalization across various architectures and datasets by finding wider, flatter optima in the loss landscape, all with negligible computational overhead.
What Problem Does This Paper Address?
Deep neural networks are typically trained using Stochastic Gradient Descent (SGD) with a decaying learning rate, aiming to find a minimum in the loss function. However, the nature of these "optima" and their relationship to generalization (how well a model performs on unseen data) is complex. While SGD finds solutions that perform well on the training data, these solutions can sometimes be "sharp" and generalize poorly. Existing methods like ensembling multiple models can improve generalization but come with increased computational cost at inference time. This paper seeks a way to achieve better generalization without sacrificing computational efficiency, by better understanding and exploiting the geometry of the loss landscape.
Methods
The core contribution is Stochastic Weight Averaging (SWA). Instead of using the final weights from a standard SGD training run, SWA collects weights from multiple points along the SGD trajectory. These points are typically sampled during the later stages of training, often when using a cyclical or constant learning rate schedule. The key idea is that these different weight snapshots, when averaged, form a single model that occupies a "wider" and "flatter" region of the loss landscape. This procedure is presented as a drop-in replacement for standard SGD training. The authors also explore the use of cyclical and constant learning rates to encourage exploration of these wider optima.
Datasets
The paper evaluates SWA on several standard computer vision datasets:
- CIFAR-10
- CIFAR-100
- ImageNet (ILSVRC-2012)
Key Insights and Ideas
- Wider Optima = Better Generalization: The paper provides strong evidence that SWA finds solutions in "wider" regions of the loss landscape compared to standard SGD. This width is hypothesized to be crucial for good generalization, as wider optima are less sensitive to small perturbations.
- Loss Landscape Shift: SWA solutions exhibit a shift in the loss function relative to SGD solutions. While SGD might converge to a point near the boundary of a flat region, SWA finds a more central point within this region, often with slightly worse training loss but significantly better test error.
- Approximation to Ensembling: SWA can be viewed as a way to approximate the benefits of model ensembling (like Fast Geometric Ensembling - FGE) but within a single model. This offers the generalization gains of ensembling without the inference-time cost.
- Exploration is Key: Using cyclical or constant learning rates during the later stages of training is crucial for SWA to explore the "periphery" of high-performing solutions and then average them to find a more robust, central point.
- Simplicity and Efficiency: SWA is remarkably easy to implement and adds almost no computational overhead to the training process.
Key Results
- Consistent Accuracy Improvements: SWA consistently outperformed standard SGD across various architectures (VGG, ResNet, DenseNet, PyramidNet, Shake-Shake) and datasets (CIFAR-10, CIFAR-100, ImageNet).
- On CIFAR-100, improvements of over 1.3% were observed.
- On ImageNet, SWA achieved 0.6-0.9% improvement for ResNet-50, ResNet-152, and DenseNet-161 with just 10 epochs of SWA training.
- Comparable to Ensembles: SWA achieved comparable or better performance than FGE ensembles (which use multiple models) with a single model.
- Wider Solutions: Visualizations and analysis showed that SWA solutions are significantly wider than those found by SGD, meaning the error increases much more slowly when moving away from the SWA optimum.
- Training from Scratch: The paper demonstrates that SWA can even be used to train models from scratch with a fixed learning rate, achieving competitive results.
Limitations and Future Work
- Initialization: While SWA can be used from scratch, the paper recommends initializing SWA from a model pre-trained with conventional training (even for a reduced number of epochs) for faster and more stable convergence.
- Learning Rate Schedules: The paper explores various learning rate schedules for SWA, suggesting that finding the optimal schedule might be task-dependent, though general guidelines are provided.
- Future Directions: The authors suggest several exciting avenues for future research, including:
- Developing SWA for faster convergence.
- Combining SWA with large batch sizes while maintaining generalization.
- Exploring SWA with stochastic MCMC methods for enhanced exploration.
- Averaging entire regions of good solutions.
Summary updated