Revisiting RCNN: On Awakening the Classification Power of Faster RCNN

Bowen Cheng, Yunchao Wei, Honghui Shi, Rogério Schmidt Feris, Jinjun Xiong, Thomas S. Huang

Introduction

Modern object detection has prioritized speed and efficiency, leading to the dominance of "end-to-end" architectures like Faster RCNN. These models achieve high performance by sharing features between classification and localization tasks. However, this efficiency comes at a hidden cost. This paper argues that the shared backbone creates a classification bottleneck, limiting the overall accuracy of state-of-the-art detectors. To address this, the authors introduce Decoupled Classification Refinement (DCR), a module designed to "awaken" a detector's potential by using a dedicated, high-capacity classifier to filter out hard false positives.

The Classification Bottleneck

In the transition from the original RCNN to Faster RCNN, the community moved toward architectures where a single backbone (e.g., ResNet) serves both the category prediction (classification) and bounding box regression (localization) branches. The authors identify three primary reasons why this shared architecture is sub-optimal:

  1. Mismatched Goals: Classification requires translation invariance (an object's identity remains the same regardless of its position), whereas localization requires translation variance (the box must shift precisely with the object). Forcing a single network to perform both leads to a compromised feature representation.
  2. Multi-task Interference: Optimizing a joint loss function for two distinct tasks often prevents either from reaching its theoretical peak performance.
  3. Receptive Field Issues: Fixed receptive fields in deep CNNs often provide excessive context for small objects and insufficient context for large ones, leading to classification errors.

Decoupled Classification Refinement (DCR)

Instead of redesigning the internal architecture of base detectors, DCR adds a specialized "correctional" stage. The process works as follows:

  • Base Detector: A standard detector (like Faster RCNN or FPN) generates initial proposals and confidence scores.
  • The DCR Module: A strong, independent classifier (e.g., ResNet-152) that shares no parameters with the base detector is introduced.
  • Image-Level Cropping: Unlike modern detectors that crop from shared feature maps (ROI Pooling), DCR crops candidate boxes directly from the original image. These crops are resized to a fixed input size (e.g., 224x224), providing an adaptive receptive field that focuses exclusively on the candidate object.
  • Hard False Positive Training: The DCR module is specifically trained on the "hard" mistakes made by the base detector, learning to distinguish between true objects and the subtle background patterns or partial overlaps that often trick standard models.

Key Insights

Analysis reveals that classification is indeed the weak link in modern detection. The authors demonstrate that if one could perfectly correct the classification of existing boxes, a Faster RCNN's mAP on PASCAL VOC could theoretically jump from ~80% to nearly 87%.

Figure 2: Analysis of false positives and their impact on mAP. (a) shows the reduction in false positives across confidence scores. (b) demonstrates how mAP improves as high-confidence false positives are filtered.

As shown in the figure above, DCR significantly reduces the number of false positives across all confidence levels. By decoupling the classification task and using image-level features, the module effectively suppresses high-confidence errors that the base detector would otherwise accept.

Performance Evaluation

The DCR module was evaluated on PASCAL VOC and MS COCO benchmarks, consistently providing significant boosts across various architectures.

Figure 4: mAP improvements across PASCAL VOC and MS COCO datasets using various baseline architectures.
  • PASCAL VOC 2007: Boosted a Faster RCNN baseline from 79.8% to 82.5% mAP.
  • PASCAL VOC 2012: Achieved 81.2% mAP, setting a new state-of-the-art for single-model detectors at the time of publication.
  • MS COCO: On the 2015 test-dev set, DCR improved a Deformable FPN baseline to 43.1% mAP, outperforming contemporary models like Mask R-CNN and RetinaNet.

Conclusion and Limitations

While DCR provides a powerful boost to accuracy by addressing the classification bottleneck, it introduces a trade-off in inference speed. Because it runs a separate deep network on multiple image crops, it is slower than standard single-stage or two-stage detectors (approximately 1.39 seconds per image on a 1080 Ti). The authors suggest that while this "brute force" decoupling is effective, future research should focus on achieving similar feature decoupling within more efficient, integrated architectures.

Summary updated

BibTeX
@inproceedings{cheng2018revisiting,
  author = {Bowen Cheng and Yunchao Wei and Honghui Shi and Rogério Schmidt Feris and Jinjun Xiong and Thomas S. Huang},
  title = {{Revisiting RCNN: On Awakening the Classification Power of Faster RCNN}},
  booktitle = {European Conference on Computer Vision},
  year = {2018},
  doi = {10.1007/978-3-030-01267-0\_28},
}
Figure