Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution

Thang Vu, Hyunjun Jang, Trung X. Pham, Chang Dong Yoo

TL;DR

Cascade RPN is a novel architecture that significantly boosts the quality of region proposals in object detection by replacing heuristic anchor settings with a multi-stage refinement process. By introducing Adaptive Convolution, the network ensures that image features stay perfectly aligned with bounding boxes as they are refined, leading to a massive 13.4-point improvement in Average Recall (AR) over the standard Region Proposal Network (RPN).

What Problem Does This Paper Address?

Most modern object detectors (like Faster R-CNN) rely on a Region Proposal Network (RPN) to identify "candidate" areas where objects might exist. However, the traditional RPN has two major "heuristic" flaws:

  1. The Anchor Problem: It relies on a manually defined set of "anchors" (pre-defined boxes of various scales and aspect ratios). If these aren't tuned perfectly for a specific dataset, performance suffers.
  2. The Alignment Problem: While researchers have tried "Iterative RPNs" (refining a box multiple times), they often ignore the fact that as a box moves and changes shape, the underlying image features used to predict it stay fixed. This spatial misalignment between the box and the features makes it difficult for the network to achieve high-quality localization.

The authors of Cascade RPN set out to build a system that learns to propose regions more systematically, without needing a library of hand-tuned anchors.

Methods

The authors propose two primary innovations to solve the issues mentioned above:

Architectural comparison of RPN variants and Cascade RPN.
  • Adaptive Convolution: This is the core technical contribution. Unlike standard convolution, which samples pixels in a rigid grid, Adaptive Convolution uses the current anchor's shape and center offset to determine where to sample features. It essentially acts as a lightweight, differentiable version of RoIAlign, ensuring that the network "looks" at the exact area covered by the refined anchor.
  • Multi-Stage Cascade with Progressive Metrics: Instead of using many anchors, Cascade RPN uses just one anchor per location. It refines this anchor over multiple stages.
  • Stage 1 uses an "anchor-free" metric (checking if the anchor center is inside an object) to handle the high variance of initial boxes.
  • Stage 2 uses a stricter "anchor-based" metric (Intersection over Union or IoU) to polish the proposals into high-quality boxes.

Datasets

The researchers evaluated their model on the COCO 2017 detection dataset, which is the gold standard for benchmarking object detection. They used the train split (115k images) for training and reported results on the val (5k images) and test-dev (20k images) splits.

Key Insights and Ideas

  • Iterative Refinement Requires Feature Adaptation: The paper demonstrates that simply repeating the regression step (Iterative RPN) yields marginal gains because the features become misaligned. The "aha moment" is that the convolution kernel itself must be transformed to match the evolving geometry of the anchor.
  • Bridging Anchor-Free and Anchor-Based: The authors found that anchor-free methods are great for getting "in the ballpark" of an object, while anchor-based methods are better for fine-tuning. By cascading them, they get the best of both worlds.
  • Efficiency Matters: Despite the multi-stage approach, Cascade RPN is highly efficient. By using a single anchor per location and a lightweight adaptive convolution, it maintains a runtime (0.06s) very close to the original RPN (0.04s).

Key Results

The performance gains reported are substantial for this field:

Performance comparison and visual example of anchor refinement.
  • Proposal Quality: Cascade RPN achieved an Average Recall (AR) of 71.7 on COCO, which is 13.4 points higher than the baseline RPN.
  • Detection Boost: When plugged into standard detectors, it improved the mean Average Precision (mAP) of Fast R-CNN by 3.1 points and Faster R-CNN by 3.5 points.
  • State-of-the-Art: It outperformed other advanced proposal methods like GA-RPN and Iterative RPN across all object scales (small, medium, and large).

Limitations and Future Work

  • Backbone Dependency: The experiments primarily used ResNet-50 with a Feature Pyramid Network (FPN). While effective, the paper doesn't explore how these gains scale with much larger or much smaller (mobile) backbones.
  • Stage Diminishing Returns: The authors noted that while two stages provide a massive boost, adding a third stage offers much smaller incremental gains (AR 71.7 to 72.2) while increasing computation time.
  • Future Directions: The success of Adaptive Convolution suggests it could be applied beyond RPNs—perhaps in the second stage of detectors or even in one-stage detectors to handle extreme aspect ratios more effectively.

Summary updated

Abstract

This paper considers an architecture referred to as Cascade Region Proposal Network (Cascade RPN) for improving the region-proposal quality and detection performance by systematically addressing the limitation of the conventional RPN that heuristically defines the anchors and aligns the features to the anchors. First, instead of using multiple anchors with predefined scales and aspect ratios, Cascade RPN relies on a single anchor per location and performs multi-stage refinement. Each stage is progressively more stringent in defining positive samples by starting out with an anchor-free metric followed by anchor-based metrics in the ensuing stages. Second, to attain alignment between the features and the anchors throughout the stages, adaptive convolution is proposed that takes the anchors in addition to the image features as its input and learns the sampled features guided by the anchors. A simple implementation of a two-stage Cascade RPN achieves 13.4 point AR higher than that of the conventional RPN, surpassing any existing region proposal methods. When adopting to Fast R-CNN and Faster R-CNN, Cascade RPN can improve the detection mAP by 3.1 and 3.5 points, respectively. The code will be made publicly available at https://github.com/thangvubk/Cascade-RPN.
BibTeX
@inproceedings{vu2019cascade,
  author = {Thang Vu and Hyunjun Jang and Trung X. Pham and Chang Dong Yoo},
  title = {{Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure