TL;DR
The authors present a remarkably simple "recipe" to boost the performance of any object detector by approximately 1.0 Average Precision (AP) on the COCO benchmark. By training a pre-trained model for an additional 12 epochs using a cyclical learning rate and averaging those 12 checkpoints, researchers can achieve significant gains with zero additional inference cost and no changes to the model architecture.
What Problem Does This Paper Address?
In recent years, progress in object detection has begun to plateau. While Average Precision on the COCO benchmark jumped from 21.9 in 2015 to over 60.0 in 2020, the "low-hanging fruit" of architectural improvements has largely been picked. Researchers now find it increasingly difficult to squeeze even a 1.0 AP improvement out of new, complex modules.
Furthermore, many performance-boosting techniques come at the cost of increased inference time or complex architectural changes. The authors sought a "free lunch"—a way to improve generalization and accuracy without making the model slower or more complicated during deployment.
Methods
The core of this work is the application of Stochastic Weight Averaging (SWA) to the specific domain of object detection. Originally proposed for general deep learning, SWA involves averaging multiple sets of weights collected at different points during the training process.
The authors' specific "recipe" is as follows:
- Take a conventionally trained object detector.
- Train it for an additional 12 epochs.
- Use a cyclical learning rate (specifically cyclical cosine annealing) where the learning rate resets every epoch.
- Save a checkpoint at the end of each of these 12 epochs.
- Average these 12 checkpoints to create the final model.
Unlike standard training, which keeps only the "best" or the final checkpoint, SWA combines the knowledge from multiple points in the weight space. Because the authors used frozen batch normalization layers (standard in many detection backbones), they avoided the need for an extra pass over the data to update batch statistics, making the process even simpler.
Datasets
The researchers conducted all experiments on the MS COCO 2017 dataset, the gold standard for object detection and instance segmentation. They trained on the train2017 split and reported results on the val2017 split using standard COCO-style Average Precision (AP) metrics.
Key Insights and Ideas
- The Geometry of Generalization: The authors argue that standard Stochastic Gradient Descent (SGD) tends to converge to solutions on the periphery of a "flat" region of low loss. These peripheral solutions often generalize poorly.
- Centering the Solution: By using a cyclical or high constant learning rate, the optimizer is forced to "bounce around" the boundaries of this optimal region. Averaging these points (W1, W2, W3 in the figure below) effectively pulls the final model (W_SWA) toward the center of the region, which typically leads to much better generalization on unseen data.
- Universality: One of the most surprising findings was that this technique works across almost all popular detector architectures—whether they are anchor-based (Faster R-CNN), anchor-free (FCOS), one-stage (RetinaNet, YOLOv3), or two-stage (Mask R-CNN).
Key Results
The results were remarkably consistent. Across various models, SWA provided a boost of ~1.0 AP:
- Mask R-CNN: Improved from 40.8 to 41.7 bbox AP.
- RetinaNet: Improved from 36.5 to 37.8 AP.
- FCOS: Improved from 39.1 to 40.2 AP.
- YOLOv3: Improved from 33.4 to 34.3 AP.
- VFNet: Improved from 52.2 to 53.4 AP.
Qualitatively, the SWA-enhanced models showed better localization and classification accuracy. As seen in the figure below, SWA models often produce fewer false positives and higher confidence scores for correct detections.
Limitations and Future Work
- Training Overhead: While there is zero cost at inference time, the technique does require an extra 12 epochs of training. For very large models or datasets, this additional compute time might be a consideration.
- Learning Rate Sensitivity: The authors found that a fixed learning rate strategy does not work well for SWA in detection; the cyclical schedule is crucial for success.
- Future Directions: The authors suggest that further investigation into different cycle lengths or even more sophisticated averaging schedules could potentially yield even higher gains. They also note that while they focused on COCO, testing this "recipe" on other challenging datasets like LVIS would be a natural next step.
Summary updated