Learning rich features at high-speed for single-shot object detection

Tiancai Wang, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Khan, Yanwei Pang, Ling Shao

TL;DR

This paper introduces LRF-Net, a single-stage object detection framework that cleverly combines the speed of pre-trained backbones with the accuracy benefits of training from scratch. It achieves this by adding a lightweight auxiliary network and a novel bi-directional feature propagation mechanism, leading to significant improvements in detecting small objects and overall accuracy while maintaining high inference speeds.

What Problem Does This Paper Address?

Single-stage object detectors are prized for their speed, making them ideal for real-time applications. However, they often struggle with accuracy, especially when detecting small objects. Traditional approaches use pre-trained backbones (like ResNet) fine-tuned for detection, which is fast but can lead to a "task gap" between classification and localization. Training from scratch can bridge this gap and improve localization, but it's computationally expensive and requires much longer training times. Existing feature pyramid networks (FPNs) primarily pass high-level semantic information downwards, neglecting the crucial flow of low- and mid-level features upwards. This paper aims to bridge the speed-accuracy trade-off by combining the strengths of both pre-training and training from scratch, while also improving feature propagation within the network.

Methods

The proposed LRF-Net builds upon the Single Shot Detector (SSD) framework. Its core innovations are:

  • Light-Weight Scratch Network (LSN): This is a parallel network trained from scratch. It takes a downsampled input image and processes it through a series of lightweight convolutional operations to extract low- and mid-level features. These features are then injected into the standard SSD prediction layers, providing complementary information that helps in detecting smaller objects.
  • Bi-directional Network: This component enhances feature circulation within the detector. It consists of two parts:
  • Bottom-up Scheme: This propagates features from earlier layers (including those from the LSN and the standard backbone) to later layers in a cascaded manner.
  • Top-down Scheme: This injects high-level semantic information from later layers back to earlier layers through independent parallel connections. This bi-directional flow ensures that both low-/mid-level and high-level semantic information are effectively utilized across all detection layers.

Datasets

The authors evaluated their approach on two datasets:

  • MS COCO: A large-scale dataset with 80 object categories, widely used for benchmarking object detection models.
  • UAVDT: A large-scale benchmark specifically designed for object detection in Unmanned Aerial Vehicle (UAV) imagery, featuring challenging scenarios like high object density and small object sizes.

Key Insights and Ideas

  • Bridging the Pre-training vs. Scratch Gap: The paper demonstrates that a hybrid approach, combining a pre-trained backbone with a small, scratch-trained auxiliary network, can achieve the best of both worlds: speed and accuracy.
  • Importance of Low-/Mid-Level Features for Small Objects: The LSN's success in improving small object detection highlights that accurate localization of small objects requires more than just high-level semantics; fine-grained local features are crucial.
  • Bi-directional Feature Flow is Key: Simply passing information top-down (as in traditional FPNs) is insufficient. The proposed bi-directional network, allowing information to flow both ways, significantly enhances the network's ability to handle multi-scale objects.
  • Efficiency Matters: The proposed LSN and bi-directional network are designed to be lightweight, ensuring that the added complexity doesn't drastically reduce inference speed.

Key Results

  • MS COCO:
    • With a VGG backbone, the proposed detector achieved an absolute gain of 7.4% in AP compared to the baseline SSD.
  • For a 300x300 input with a ResNet backbone, the detector achieved 34.3 AP, surpassing existing single-stage methods while running at 91 FPS on a single Titan X GPU.
  • The detector significantly improved performance on small objects, more than doubling the detection rate compared to the baseline SSD.
  • UAVDT:
    • The detector achieved an AP score of 37.81, outperforming the two-stage R-FCN (34.35 AP) while providing a nearly 20-fold speedup.
  • Qualitative Results: Figure 5 showcases impressive detection results on both COCO and UAVDT datasets, particularly highlighting the accurate localization of small objects in challenging scenes.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of object detection (e.g., extreme occlusion, very low resolution). The performance gains, while significant, are presented in comparison to specific baselines and state-of-the-art methods at the time of publication.
  • Future Work: The authors suggest that their framework can be extended to other single-stage detectors and that further exploration of the bi-directional network's design choices could yield even better results. The code for their LRF-Net is made publicly available, encouraging further research and development.

Summary updated

Abstract

Single-stage object detection methods have received significant attention recently due to their characteristic realtime capabilities and high detection accuracies. Generally, most existing single-stage detectors follow two common practices: they employ a network backbone that is pretrained on ImageNet for the classification task and use a top-down feature pyramid representation for handling scale variations. Contrary to common pre-training strategy, recent works have demonstrated the benefits of training from scratch to reduce the task gap between classification and localization, especially at high overlap thresholds. However, detection models trained from scratch require significantly longer training time compared to their typical finetuning based counterparts. We introduce a single-stage detection framework that combines the advantages of both fine-tuning pre-trained models and training from scratch. Our framework constitutes a standard network that uses a pre-trained backbone and a parallel light-weight auxiliary network trained from scratch. Further, we argue that the commonly used top-down pyramid representation only focuses on passing high-level semantics from the top layers to bottom layers. We introduce a bi-directional network that efficiently circulates both low-/mid-level and high-level semantic information in the detection framework. Experiments are performed on MS COCO and UAVDT datasets. Compared to the baseline, our detector achieives an absolute gain of 7.4% and 4.2% in average precision (AP) on MS COCO and UAVDT datasets, respectively using VGG backbone. For a 300×300 input on the MS COCO test set, our detector with ResNet backbone surpasses existing single-stage detection methods for single-scale inference achieving 34.3 AP, while operating at an inference time of 19 milliseconds on a single Titan X GPU. Code is available at https://github.com/vaesl/LRF-Net.
BibTeX
@inproceedings{wang2019learning,
  author = {Tiancai Wang and Rao Muhammad Anwer and Hisham Cholakkal and Fahad Khan and Yanwei Pang and Ling Shao},
  title = {{Learning rich features at high-speed for single-shot object detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=HClblxs5919},
}
Figure