R-FCN: Object Detection via Region-based Fully Convolutional Networks

Jifeng Dai, Yi Li, Kaiming He, Jian Sun

Abstract

We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20x faster than the Faster R-CNN counterpart.

Introduction

In the development of deep learning for computer vision, a fundamental dilemma exists between image classification and object detection. Image classification favors translation invariance: the identity of an object (e.g., a cat) remains the same regardless of its position in the frame. Modern fully convolutional architectures like ResNets are engineered to maximize this invariance.

Conversely, object detection requires translation variance: the system must be sensitive to an object’s precise spatial coordinates to generate accurate bounding boxes. Previous state-of-the-art detectors, such as Faster R-CNN, addressed this by partitioning the network into a shared convolutional subnetwork and a "heavy" per-region subnetwork. While effective, this approach is computationally expensive because the per-region subnetwork must be executed for every candidate region (often hundreds per image). R-FCN (Region-based Fully Convolutional Network) aims to resolve this by making the entire network fully convolutional, thereby sharing nearly all computation across the image.

Methodology

The core innovation of R-FCN is the introduction of position-sensitive score maps and a corresponding position-sensitive RoI pooling layer.

Overview of the R-FCN architecture showing position-sensitive score maps and RoI pooling.

Instead of a single score map per class, R-FCN generates a bank of $k^2$ score maps for each class (e.g., $3 \times 3 = 9$). Each map is trained to respond to a specific relative position, such as "top-left" or "bottom-right" of an object. When evaluating a Region of Interest (RoI), the network divides it into a $k \times k$ grid. The RoI pooling layer then selectively samples from the corresponding score map for each grid bin—for example, the top-left bin only pools from the top-left score map. Finally, these $k^2$ values are averaged (voted) to produce the final classification score. This mechanism allows the network to maintain spatial sensitivity while remaining fully convolutional.

Experimental Results

R-FCN was evaluated on the PASCAL VOC 2007/2012 and MS COCO datasets using a ResNet-101 backbone.

  • Accuracy: On PASCAL VOC 2007, R-FCN achieved a mean Average Precision (mAP) of 83.6%, matching the performance of the best region-based detectors.
  • Efficiency: The model achieved an inference speed of 170ms per image on an NVIDIA K40 GPU. This represents a speedup of 2.5x to 20x compared to Faster R-CNN implementations using the same backbone, primarily due to the elimination of the heavy per-region subnetwork.
  • COCO Performance: On the MS COCO benchmark, R-FCN achieved 53.2% [email protected], demonstrating strong performance on a large-scale dataset with 80 categories.

Discussion and Conclusion

The success of R-FCN stems from the realization that translation-invariant networks can be made translation-variant by forcing output channels to specialize in specific spatial regions. This "shepherding" of features allows for a highly efficient architecture where the cost per region proposal is negligible.

Visualization of a position-sensitive score map responding to the center of an object.

While R-FCN is highly efficient, challenges remain in detecting very small objects, a common issue for fully convolutional networks due to feature map downsampling. However, the simplicity and speed of R-FCN make it a powerful framework for real-time object detection and a foundation for unifying various vision tasks under a fully convolutional paradigm.

Summary updated

Abstract

We present region-based, fully convolutional networks for accurate and efficient object detection. In contrast to previous region-based detectors such as Fast/Faster R-CNN that apply a costly per-region subnetwork hundreds of times, our region-based detector is fully convolutional with almost all computation shared on the entire image. To achieve this goal, we propose position-sensitive score maps to address a dilemma between translation-invariance in image classification and translation-variance in object detection. Our method can thus naturally adopt fully convolutional image classifier backbones, such as the latest Residual Networks (ResNets), for object detection. We show competitive results on the PASCAL VOC datasets (e.g., 83.6% mAP on the 2007 set) with the 101-layer ResNet. Meanwhile, our result is achieved at a test-time speed of 170ms per image, 2.5-20x faster than the Faster R-CNN counterpart. Code is made publicly available at: https://github.com/daijifeng001/r-fcn
BibTeX
@inproceedings{dai2016r,
  author = {Jifeng Dai and Yi Li and Kaiming He and Jian Sun},
  title = {{R-FCN: Object Detection via Region-based Fully Convolutional Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2016},
  url = {https://openreview.net/forum?id=S1bknwZuWS},
}
Figure