RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles

Ramin Nabati, Hairong Qi

TL;DR

RRPN (Radar Region Proposal Network) is a real-time algorithm that uses Radar data to generate object proposals for two-stage image-based detectors. By replacing computationally expensive vision-based proposal methods with Radar-guided anchors, the system achieves a 100x speedup and higher detection accuracy on the NuScenes dataset compared to traditional methods like Selective Search.

What Problem Does This Paper Address?

In the world of autonomous driving, "two-stage" object detectors (like Fast R-CNN) are highly accurate but notoriously slow. The bottleneck is the Region Proposal Network (RPN)—the first stage that scans the entire image to "guess" where objects might be. Standard vision-based RPNs are computationally heavy, often making them unsuitable for the split-second decision-making required in self-driving cars.

While most vehicles are equipped with both cameras and Radars, these sensors are often used independently or fused late in the pipeline. Radar is excellent at determining distance and velocity but terrible at classifying what an object is. Cameras are the opposite. This paper addresses the "speed vs. accuracy" trade-off by using Radar’s spatial strengths to tell the camera exactly where to look, bypassing the slow vision-based search entirely.

Methods

The authors propose a three-step pipeline to turn raw Radar points into high-quality image proposals:

  1. Perspective Transformation: Radar detections are typically reported in a "bird’s eye view" (distance and azimuth). The RRPN maps these 3D points onto the 2D image coordinate system using a projection matrix derived from camera calibration.
  2. Adaptive Anchor Generation: Simply placing a box on a Radar point isn't enough because Radar points don't always hit the exact center of an object. RRPN generates a set of "anchor boxes" with different aspect ratios and sizes. Crucially, it also generates translated versions (shifted left, right, or down) to ensure the proposal captures the full extent of the object even if the Radar return is off-center.
  3. Distance Compensation: In an image, objects get smaller as they get further away. RRPN uses the Radar’s precise range data ($d_i$) to scale the anchor boxes dynamically. They use a learned linear relationship ($S_i = \alpha \frac{1}{d_i} + \beta$) to determine the scaling factor, ensuring that proposals for distant cars are appropriately tiny while nearby ones are large.

Datasets

The researchers evaluated RRPN using the NuScenes dataset, which was the first large-scale public dataset to provide synchronized and annotated data from a full sensor suite (6 cameras, 5 Radars, LiDAR, etc.).

  • They created two subsets: NS-F (front-facing data only) and NS-FB (front and back data).
  • The dataset includes 25 object classes, which the authors merged into 6 main categories: Car, Truck, Person, Motorcycle, Bicycle, and Bus.

Key Insights and Ideas

  • Radar as an Attention Mechanism: Instead of the neural network "sweating" over every pixel to find a pedestrian, the Radar acts as a pointer. This "attention" allows the system to ignore irrelevant parts of the image (like clouds or billboards) and focus computational power only on physical objects.
  • Bypassing the "Black Box": Unlike traditional RPNs that require extensive training to learn how to propose regions, RRPN uses the physics of the sensors. The distance-to-scale mapping is a simple, interpretable geometric solution.
  • Robustness to Occlusion: Because Radar can "see" through some materials or detect objects based on reflections that cameras might miss, the RRPN-generated proposals are often more accurate than vision-only proposals in crowded or high-overlap scenarios.

Key Results

  • Massive Speedup: RRPN operates more than 100x faster than the Selective Search algorithm. While Selective Search takes 2–7 seconds per image, RRPN processes 70–90 images per second.
  • Improved Accuracy: When plugged into a Fast R-CNN framework, RRPN achieved higher Mean Average Precision (mAP) and Mean Average Recall (mAR) than the baseline.
  • Class-Specific Gains: The method showed significant improvements in detecting smaller or more "difficult" classes like Persons, Motorcycles, and Bicycles, where traditional vision-based proposals often struggle.
  • Backbone Flexibility: The authors demonstrated success using both ResNet-101 and ResNeXt-101 backbones, showing the method is robust across different neural architectures.

Limitations and Future Work

  • Radar Noise: The paper notes that Radars can sometimes produce "ghost" detections or noise. While RRPN filters some of this, a high false-positive rate from the Radar could lead to unnecessary image processing.
  • Vertical Localization: Radar data used in this study is 2D (range and azimuth). It doesn't provide height information, which is why the authors had to include "bottom-aligned" anchors to ensure the boxes sit correctly on the road surface.
  • Sensor Synchronization: The method relies heavily on perfect calibration and synchronization between the Radar and Camera. Any misalignment in the projection matrix would cause the proposals to miss the objects entirely.
  • Future Direction: The authors suggest that further refining the anchor generation logic to account for specific object orientations (using Radar's velocity data) could further improve precision.

Summary updated

Abstract

Region proposal algorithms play an important role in most state-of-the-art two-stage object detection networks by hypothesizing object locations in the image. Nonetheless, region proposal algorithms are known to be the bottleneck in most two-stage object detection networks, increasing the processing time for each image and resulting in slow networks not suitable for real-time applications such as autonomous driving vehicles. In this paper we introduce RRPN, a Radar-based real-time region proposal algorithm for object detection in autonomous driving vehicles. RRPN generates object proposals by mapping Radar detections to the image coordinate system and generating pre-defined anchor boxes for each mapped Radar detection point. These anchor boxes are then transformed and scaled based on the object's distance from the vehicle, to provide more accurate proposals for the detected objects. We evaluate our method on the newly released NuScenes dataset [1] using the Fast R-CNN object detection network [2]. Compared to the Selective Search object proposal algorithm [3], our model operates more than 100× faster while at the same time achieves higher detection precision and recall. Code has been made publicly available at https://github.com/mrnabati/RRPN.
BibTeX
@inproceedings{nabati2019rrpn,
  author = {Ramin Nabati and Hairong Qi},
  title = {{RRPN: Radar Region Proposal Network for Object Detection in Autonomous Vehicles}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=bRWGzS5TJb},
}

Loading related content...

Figure