SWA Object Detection

Haoyang Zhang, Ying Wang, Feras Dayoub, Niko SünderhaufarXiv 2020 · arXiv

TL;DR

The authors present a remarkably simple "recipe" to boost the performance of any object detector by approximately 1.0 Average Precision (AP) on the COCO benchmark. By training a pre-trained model for an additional 12 epochs using a cyclical learning rate and averaging those 12 checkpoints, researchers can achieve significant gains with zero additional inference cost and no changes to the model architecture.

What Problem Does This Paper Address?

In recent years, progress in object detection has begun to plateau. While Average Precision on the COCO benchmark jumped from 21.9 in 2015 to over 60.0 in 2020, the "low-hanging fruit" of architectural improvements has largely been picked. Researchers now find it increasingly difficult to squeeze even a 1.0 AP improvement out of new, complex modules.

Furthermore, many performance-boosting techniques come at the cost of increased inference time or complex architectural changes. The authors sought a "free lunch"—a way to improve generalization and accuracy without making the model slower or more complicated during deployment.

Methods

The core of this work is the application of Stochastic Weight Averaging (SWA) to the specific domain of object detection. Originally proposed for general deep learning, SWA involves averaging multiple sets of weights collected at different points during the training process.

The authors' specific "recipe" is as follows:

  1. Take a conventionally trained object detector.
  2. Train it for an additional 12 epochs.
  3. Use a cyclical learning rate (specifically cyclical cosine annealing) where the learning rate resets every epoch.
  4. Save a checkpoint at the end of each of these 12 epochs.
  5. Average these 12 checkpoints to create the final model.
Figure 2: Illustration of cyclical cosine annealing learning rates. In each cycle, the learning rate decreases at each iter- ation from the initial learning rate lrmax (0.02 in this case) to the ending learning rate lrmin (0.0002 in this case).

Unlike standard training, which keeps only the "best" or the final checkpoint, SWA combines the knowledge from multiple points in the weight space. Because the authors used frozen batch normalization layers (standard in many detection backbones), they avoided the need for an extra pass over the data to update batch statistics, making the process even simpler.

Datasets

The researchers conducted all experiments on the MS COCO 2017 dataset, the gold standard for object detection and instance segmentation. They trained on the train2017 split and reported results on the val2017 split using standard COCO-style Average Precision (AP) metrics.

Key Insights and Ideas

  • The Geometry of Generalization: The authors argue that standard Stochastic Gradient Descent (SGD) tends to converge to solutions on the periphery of a "flat" region of low loss. These peripheral solutions often generalize poorly.
  • Centering the Solution: By using a cyclical or high constant learning rate, the optimizer is forced to "bounce around" the boundaries of this optimal region. Averaging these points (W1, W2, W3 in the figure below) effectively pulls the final model (W_SWA) toward the center of the region, which typically leads to much better generalization on unseen data.
Figure 1: Illustration of SWA. W1, W2 and W3 represent different SGD solutions. WSW A indicates the SWA solu- tion that is the average of those SGD solutions.
  • Universality: One of the most surprising findings was that this technique works across almost all popular detector architectures—whether they are anchor-based (Faster R-CNN), anchor-free (FCOS), one-stage (RetinaNet, YOLOv3), or two-stage (Mask R-CNN).

Key Results

The results were remarkably consistent. Across various models, SWA provided a boost of ~1.0 AP:

  • Mask R-CNN: Improved from 40.8 to 41.7 bbox AP.
  • RetinaNet: Improved from 36.5 to 37.8 AP.
  • FCOS: Improved from 39.1 to 40.2 AP.
  • YOLOv3: Improved from 33.4 to 34.3 AP.
  • VFNet: Improved from 52.2 to 53.4 AP.

Qualitatively, the SWA-enhanced models showed better localization and classification accuracy. As seen in the figure below, SWA models often produce fewer false positives and higher confidence scores for correct detections.

Figure 3: Comparative qualitative examples. Left: detection results of pre-trained models. Right: detection results of corresponding SWA models. The comparison shows that SWA can improve both the object localization and classification accuracy, resulting in less false positives and higher recall rates.

Limitations and Future Work

  • Training Overhead: While there is zero cost at inference time, the technique does require an extra 12 epochs of training. For very large models or datasets, this additional compute time might be a consideration.
  • Learning Rate Sensitivity: The authors found that a fixed learning rate strategy does not work well for SWA in detection; the cyclical schedule is crucial for success.
  • Future Directions: The authors suggest that further investigation into different cycle lengths or even more sophisticated averaging schedules could potentially yield even higher gains. They also note that while they focused on COCO, testing this "recipe" on other challenging datasets like LVIS would be a natural next step.

Summary updated

Abstract

Do you want to improve 1.0 AP for your object detector without any inference cost and any change to your detector? Let us tell you such a recipe. It is surprisingly simple: train your detector for an extra 12 epochs using cyclical learning rates and then average these 12 checkpoints as your final detection model}. This potent recipe is inspired by Stochastic Weights Averaging (SWA), which is proposed in arXiv:1803.05407 for improving generalization in deep neural networks. We found it also very effective in object detection. In this technique report, we systematically investigate the effects of applying SWA to object detection as well as instance segmentation. Through extensive experiments, we discover the aforementioned workable policy of performing SWA in object detection, and we consistently achieve $\sim$1.0 AP improvement over various popular detectors on the challenging COCO benchmark, including Mask RCNN, Faster RCNN, RetinaNet, FCOS, YOLOv3 and VFNet. We hope this work will make more researchers in object detection know this technique and help them train better object detectors. Code is available at: https://github.com/hyz-xmaster/swa_object_detection .
BibTeX
@misc{zhang2020swa,
  author = {Haoyang Zhang and Ying Wang and Feras Dayoub and Niko Sünderhauf},
  title = {{SWA Object Detection}},
  year = {2020},
  eprint = {2012.12645},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.12645},
}
Figure