Timely Fusion of Surround Radar/Lidar for Object Detection in Autonomous Driving Systems

Wenjing Xie, Tao Hu, Neiwen Ling, Guoliang Xing, Shaoshan Liu, Nan Guan

Abstract

Fusing Radar and Lidar sensor data can fully utilize their complementary advantages and provide more accurate reconstruction of the surrounding for autonomous driving systems. Surround Radar/Lidar can provide 360-degree view sampling with the minimal cost, which are promising sensing hardware solutions for autonomous driving systems. However, due to the intrinsic physical constraints, the rotating speed of surround Radar, and thus the frequency to generate Radar data frames, is much lower than surround Lidar. Existing Radar/Lidar fusion methods have to work at the low frequency of surround Radar, which cannot meet the high responsiveness requirement of autonomous driving systems. This paper develops techniques to fuse surround Radar/Lidar with working frequency only limited by the faster surround Lidar instead of the slower surround Radar, based on the state-of-the-art object detection model MVDNet. The basic idea of our approach is simple: we let MVDNet work with temporally unaligned data from Radar/Lidar, so that fusion can take place at any time when a new Lidar data frame arrives, instead of waiting for the slow Radar data frame. However, directly applying MVDNet to temporally unaligned Radar/Lidar data greatly degrades its object detection accuracy. The key information revealed in this paper is that we can achieve high output frequency with little accuracy loss by enhancing the training procedure to explore the temporal redundancy in MVDNet so that it can tolerate the temporal unalignment of input data. We explore several different ways of training enhancement and compare them quantitatively with experiments.

Introduction

In the development of autonomous vehicles, robust perception of the environment is paramount. Lidar and Radar are the primary sensors used for this task, offering complementary strengths: Lidar provides high-resolution 360° point clouds, while Radar is capable of operating in adverse weather conditions where Lidar might fail. Fusing these two modalities is essential for safe navigation.

However, a significant challenge arises from the physical constraints of these sensors. Surround sensors rotate to scan their environment, and because Radar utilizes millimeter waves, it typically rotates much slower than Lidar. For example, a standard Lidar sensor might produce 20 frames per second (20Hz), whereas a Radar sensor might only produce 4 frames per second (4Hz).

Temporal misalignment between Radar and Lidar frames.

As illustrated in Figure 1, this discrepancy creates a temporal synchronization bottleneck. Traditional fusion methods typically wait for the slower Radar frame to arrive before performing calculations, limiting the system's update frequency to 4Hz. This 250ms latency is far too slow for high-speed driving scenarios. Attempting to fuse the latest Lidar frame with a "stale" Radar frame using standard models leads to a significant drop in detection accuracy.

Methodology

The proposed approach, built upon the MVDNet architecture, aims to decouple fusion frequency from the slower Radar hardware.

MVDNet Architecture

MVDNet is a multi-view fusion network designed for Radar and Lidar data. It processes both sensor streams through dedicated feature extractors before combining them into a unified representation.

The MVDNet architecture for Radar and Lidar stream fusion.

As shown in Figure 3, the architecture consists of the Multi-View Detection Region Proposal Network (MVD-RPN) and the Multi-View Detection Refinement Network (MVD-RFN). Features are extracted independently from each stream, fused at the proposal stage, and then further refined through temporal fusion to ensure consistent object tracking over time.

Timely Fusion and Offset-Aware Training

To achieve "Timely Fusion," the system triggers the fusion process whenever a new Lidar frame arrives, pairing it with the most recent available Radar data. To address the resulting temporal offset, the authors proposed two training enhancements:

  1. Separate Training Strategy: This strategy involves training specific model branches for different levels of timing offset. A multi-branch architecture then selects the appropriate fusion logic based on the "staleness" (index) of the Radar data.
  2. Mixed Training Strategy: A single model is trained on a diverse mix of both aligned and unaligned data, forcing the neural network to become robust to temporal shifts.

Historical Frame Skipping

MVDNet utilizes historical frames to improve detection accuracy. The researchers found that by "skipping" every other frame while looking further back in time, they could maintain the same temporal context with half the computational workload, significantly reducing latency without sacrificing performance.

Experimental Results

The study utilized the Oxford Radar RobotCar (ORR) dataset, which features a 4Hz Radar and 20Hz Lidars.

Performance Metrics

  • Frequency Boost: The system successfully increased the fusion frequency from 4Hz to 20Hz, matching the Lidar's peak performance.
  • Accuracy Recovery: Without the proposed training, using unaligned data caused Average Precision (AP) at IoU 0.8 to drop from 0.735 to 0.486. With the Separate Training strategy, the model recovered nearly all of this loss, achieving an AP of 0.722.
  • Efficiency: The Frame Skipping method reduced the number of required historical frames from 4 to 2 while maintaining comparable accuracy, optimizing the detection pipeline for real-time use.

Conclusion

This paper demonstrates that high-frequency sensor fusion is possible even with mismatched hardware rates. By leveraging temporal redundancy and implementing offset-aware training, autonomous systems can achieve high-responsiveness detection without sacrificing accuracy. Future work will explore the impact of these methods on smaller, faster-moving objects like cyclists and pedestrians, and aim to optimize the architecture for low-power embedded systems.

Summary updated

Abstract

Fusing Radar and Lidar sensor data can fully utilize their complementary advantages and provide more accurate reconstruction of the surrounding for autonomous driving systems. Surround Radar/Lidar can provide 360-degree view sampling with the minimal cost, which are promising sensing hardware solutions for autonomous driving systems. However, due to the intrinsic physical constraints, the rotating speed of surround Radar, and thus the frequency to generate Radar data frames, is much lower than surround Lidar. Existing Radar/Lidar fusion methods have to work at the low frequency of surround Radar, which cannot meet the high responsiveness requirement of autonomous driving systems.This paper develops techniques to fuse surround Radar/Lidar with working frequency only limited by the faster surround Lidar instead of the slower surround Radar, based on the state-of-the-art object detection model MVDNet. The basic idea of our approach is simple: we let MVDNet work with temporally unaligned data from Radar/Lidar, so that fusion can take place at any time when a new Lidar data frame arrives, instead of waiting for the slow Radar data frame. However, directly applying MVDNet to temporally unaligned Radar/Lidar data greatly degrades its object detection accuracy. The key information revealed in this paper is that we can achieve high output frequency with little accuracy loss by enhancing the training procedure to explore the temporal redundancy in MVDNet so that it can tolerate the temporal unalignment of input data. We explore several different ways of training enhancement and compare them quantitatively with experiments.
BibTeX
@inproceedings{xie2024timely,
  author = {Wenjing Xie and Tao Hu and Neiwen Ling and Guoliang Xing and Shaoshan Liu and Nan Guan},
  title = {{Timely Fusion of Surround Radar/Lidar for Object Detection in Autonomous Driving Systems}},
  booktitle = {RTCSA},
  year = {2024},
  doi = {10.1109/rtcsa62462.2024.00014},
}
Figure