Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution

Thang Vu, Hyunjun Jang, Trung X. Pham, Chang D. Yoo

TL;DR

This paper introduces Cascade RPN, a novel region proposal network that improves object detection by systematically addressing limitations in how anchors and features are handled. By using a multi-stage refinement process with adaptive convolutions, Cascade RPN significantly boosts the quality of region proposals, leading to better overall object detection performance.

What Problem Does This Paper Address?

Object detection, a crucial task in computer vision, relies heavily on accurately identifying potential object locations (region proposals) before classifying them. The dominant approach, the Region Proposal Network (RPN), uses predefined "anchor boxes" of various scales and aspect ratios to generate these proposals. However, this method has limitations:

  • Heuristic Anchor Design: The predefined anchors require careful tuning and may not always perfectly match the diverse shapes and sizes of objects in real-world scenes.
  • Feature-Anchor Misalignment: As anchors are refined through multiple stages, the corresponding image features can become misaligned, hindering accurate localization. Conventional RPNs rely on heuristics to maintain this alignment, which can be suboptimal.

This paper aims to overcome these limitations by proposing a more systematic approach to region proposal generation that ensures better alignment between features and anchors throughout the refinement process.

Methods

The core innovation is the Cascade Region Proposal Network (Cascade RPN), which tackles the aforementioned problems through two main mechanisms:

  1. Multi-Stage Refinement with Progressive Sample Discrimination: Instead of relying on a large set of predefined anchors, Cascade RPN uses a single anchor per location and refines it over multiple stages. Crucially, each stage employs increasingly stringent criteria for classifying a proposal as "positive" (likely containing an object). It starts with an anchor-free metric (e.g., anchor center within an object's center region) in the first stage and transitions to anchor-based metrics (e.g., Intersection over Union - IoU) in subsequent stages. This progressive approach helps to generate higher-quality proposals.
  2. Adaptive Convolution: To maintain alignment between the refined anchors and the image features across stages, Cascade RPN introduces adaptive convolution. Unlike standard convolutions that sample features on a fixed grid, adaptive convolution learns to sample features based on the current anchor's position and shape. This is achieved by inferring an offset field from the anchor, allowing the convolution to dynamically adapt its sampling locations. This acts as a lightweight, learnable alignment mechanism, similar in spirit to RoIAlign but integrated directly into the RPN.

Datasets

The paper uses the COCO 2017 detection dataset for training and evaluation.

  • Training was performed on the train split (115k images).
  • Region proposal performance and ablation studies were reported on the val split (5k images).
  • Benchmarking detection performance was reported on the test-dev split (20k images).

No new benchmarks were introduced.

Key Insights and Ideas

  • Alignment is Key: The paper strongly emphasizes that maintaining alignment between anchor boxes and image features is critical for high-quality region proposals, and that heuristic approaches are insufficient.
  • Progressive Refinement Works: Using a multi-stage approach with increasingly strict positive sample criteria effectively filters out low-quality proposals and refines high-quality ones.
  • Adaptive Convolution as a Solution: The proposed adaptive convolution is a clever way to dynamically adjust feature sampling to match evolving anchor shapes, directly addressing the alignment problem.
  • Anchor-Free + Anchor-Based Synergy: Combining anchor-free and anchor-based metrics for sample discrimination across stages proves more effective than using either alone, highlighting a synergistic benefit.

Key Results

  • Region Proposal Performance: A simple two-stage Cascade RPN achieved an Average Recall (AR) of 61.1 on the COCO 2017 validation set, which is 13.4 points higher than the conventional RPN (44.6 AR100). This significantly surpassed existing region proposal methods.
  • Object Detection Performance: When integrated into two-stage detectors:
  • Cascade RPN improved Fast R-CNN's mAP by 3.1 points (from 37.0 to 40.1).
  • Cascade RPN improved Faster R-CNN's mAP by 3.5 points (from 37.1 to 40.6).
  • Efficiency: The proposed method adds minimal computational overhead, with a two-stage Cascade RPN achieving its best results with only a slight increase in inference time (0.06s vs. 0.04s for RPN).

Limitations and Future Work

  • Number of Stages: While a 3-stage Cascade RPN showed slightly better performance (72.2 AR1000) than the 2-stage version (71.7 AR1000), the paper suggests the 2-stage Cascade RPN offers the best trade-off between performance and inference time. This implies that further increasing stages might not yield proportional benefits.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success of Cascade RPN suggests potential applications in other tasks requiring precise localization and feature alignment, such as instance segmentation or keypoint detection. The adaptive convolution mechanism itself could also be explored in broader convolutional network architectures.

Summary updated

Abstract

This paper considers an architecture referred to as Cascade Region Proposal Network (Cascade RPN) for improving the region-proposal quality and detection performance by systematically addressing the limitation of the conventional RPN that heuristically defines the anchors and aligns the features to the anchors. First, instead of using multiple anchors with predefined scales and aspect ratios, Cascade RPN relies on a single anchor per location and performs multi-stage refinement. Each stage is progressively more stringent in defining positive samples by starting out with an anchor-free metric followed by anchor-based metrics in the ensuing stages. Second, to attain alignment between the features and the anchors throughout the stages, adaptive convolution is proposed that takes the anchors in addition to the image features as its input and learns the sampled features guided by the anchors. A simple implementation of a two-stage Cascade RPN achieves 13.4 point AR higher than that of the conventional RPN, surpassing any existing region proposal methods. When adopting to Fast R-CNN and Faster R-CNN, Cascade RPN can improve the detection mAP by 3.1 and 3.5 points, respectively. The code will be made publicly available at https://github.com/thangvubk/Cascade-RPN.
BibTeX
@inproceedings{vu2019cascade,
  author = {Thang Vu and Hyunjun Jang and Trung X. Pham and Chang D. Yoo},
  title = {{Cascade RPN: Delving into High-Quality Region Proposal Network with Adaptive Convolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure