A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data

Kavin Chandrasekaran, Sorin Grigorescu, Gijs Dubbelman, Pavol Jancura

Introduction

Autonomous Driving Systems (ADS) rely on a multi-modal sensor suite, typically including cameras, LiDAR, and radar. While cameras offer rich semantic information, they are susceptible to adverse weather and lack inherent depth perception. Radar, conversely, is robust to environmental conditions but is often processed into sparse point clouds, discarding valuable contextual data.

A significant challenge in sensor fusion is the geometric discrepancy between camera perspective views and radar's polar (Range-Azimuth) coordinate system. Many current solutions employ computationally intensive transformers to bridge this gap, which can be prohibitive for real-time edge deployment. This paper introduces REFNet, a resource-efficient fusion network that pre-aligns camera data into a polar Bird’s-Eye View (BEV) to match the radar's natural domain, achieving high accuracy at nearly 60 FPS.

Methodology

The core innovation of REFNet lies in its "pre-alignment" strategy, which simplifies the fusion task for the neural network.

Sensor Alignment and Processing

Instead of requiring the network to learn complex coordinate transformations, the authors implement a geometric preprocessing pipeline:

  1. Camera Stream: Perspective images are projected into a Cartesian BEV and subsequently transformed into a BEV Polar domain. This ensures that camera features are spatially aligned with the radar's Range-Azimuth format before feature extraction.
  2. Radar Stream: The network processes raw Range-Doppler (RD) tensors. A channel-swapping strategy is utilized to recover angular information, converting RD data into Range-Azimuth (RA) features.
The REFNet architecture features an image processing pipeline that transforms camera data into the polar domain to match raw radar features before fusion.

Architecture

REFNet utilizes a dual-stream architecture based on ResNet-50 encoders. One branch processes the polar-transformed camera images, while the other handles the radar data. The resulting features from both decoders are fused via channel concatenation. A specialized detection head then performs classification and regression to localize vehicles in the range-azimuth space.

Experimental Setup

The model was evaluated using the RADIal dataset, which provides 2 hours of synchronized data from automotive-grade sensors. Crucially, RADIal includes raw Analog-to-Digital Converter (ADC) signals and Range-Doppler views, enabling the study of low-level radar fusion. The dataset contains approximately 25,000 frames, with over 8,000 labeled frames for vehicle detection.

Key Results

REFNet demonstrates a superior balance of localization precision and computational efficiency.

  • Accuracy: On the RADIal dataset, REFNet achieved state-of-the-art localization with a Range Error of 0.11m and an Angle Error of 0.09°. It reached a competitive 93.49% F1-score.
  • Efficiency: The model operates at 58.91 FPS on an NVIDIA RTX A6000. It maintains a significantly smaller memory footprint than comparable models like EchoFusion, using nearly four times fewer parameters.
  • Ablation Study: The authors confirmed that using Polar BEV input for the camera stream (93.49% F1) significantly outperforms Cartesian BEV input (88.41% F1), validating the importance of domain alignment.
The BEV Polar representation maps visual features into a range (r) and azimuth (theta) coordinate system, facilitating direct fusion with radar data.

Conclusion

By adopting the polar coordinate system as a "common language" for sensor fusion, REFNet eliminates the need for heavy transformer-based cross-attention mechanisms. This approach preserves the rich information in raw radar data while maintaining the real-time performance necessary for autonomous driving. Future work will focus on expanding detection to more diverse object classes and improving robustness to camera calibration errors.

Summary updated

BibTeX
@inproceedings{chandrasekaran2024resource,
  author = {Kavin Chandrasekaran and Sorin Grigorescu and Gijs Dubbelman and Pavol Jancura},
  title = {{A Resource Efficient Fusion Network for Object Detection in Bird's-Eye View using Camera and Raw Radar Data}},
  booktitle = {ITSC},
  year = {2024},
  doi = {10.1109/itsc58415.2024.10919729},
}
Figure