Timely Fusion of Surround Radar/Lidar for Object Detection in Autonomous Driving Systems

Wenjing Xie, Tao Hu, Neiwen Ling, Guoliang Xing, Chun Jason Xue, Nan Guan

TL;DR

Autonomous vehicles often fuse Lidar and Radar data for robust perception, but they face a "bottleneck" because Lidar sensors rotate much faster (20Hz) than Radar (4Hz). This paper introduces a method to perform fusion at the faster Lidar frequency by training the model to handle temporally unaligned data, achieving high-responsiveness detection with almost no loss in accuracy.

What Problem Does This Paper Address?

In the world of autonomous driving, Lidar and Radar are the "eyes" of the car. Lidar provides high-resolution 360° point clouds but fails in fog or snow; Radar is lower resolution but pierces through bad weather. Fusing them is essential for safety.

However, there is a physical synchronization problem. Surround sensors rotate to scan the environment. Because Radar uses millimeter waves (which travel slower than Lidar’s light), it typically rotates much slower. For example, a standard Lidar might produce 20 frames per second (20Hz), while a Radar only produces 4.

Existing fusion methods usually wait for the slow Radar frame to arrive before performing a calculation. This means the car’s "brain" only updates its fused view every 250ms—far too slow for high-speed driving where every millisecond counts. If you try to fuse the latest Lidar frame with a "stale" Radar frame, standard models break down, and detection accuracy plummets.

Methods

The researchers built their solution on MVDNet, a popular deep-learning model for Radar/Lidar fusion. Their core strategy is to decouple the fusion frequency from the slow Radar hardware.

  1. Timely Fusion: Instead of waiting for a new Radar frame, the system triggers fusion the moment a new Lidar frame arrives, pairing it with the most recent (potentially "stale") Radar data.
  2. Offset-Aware Training: To fix the accuracy drop caused by this timing mismatch (the "offset"), the authors proposed two training enhancements:
  3. Separate Training Strategy: They trained specific "branches" of the model for different levels of timing offset. A multi-branch architecture then selects the right fusion logic based on how "stale" the current Radar data is.
  4. Mixed Training Strategy: They trained a single model on a diverse mix of aligned and unaligned data, forcing the neural network to become robust to temporal shifts.
  5. Historical Frame Skipping: MVDNet uses several past frames to improve detection. The authors found that by "skipping" every other frame but looking further back in time, they could maintain the same amount of temporal context with half the computational workload.

Datasets

The study utilized the Oxford Radar RobotCar (ORR) dataset, a gold standard for multi-modal autonomous driving research.

  • Hardware: It features a NavTech CTS350-X Radar (4Hz) and two Velodyne HDL-32E Lidars (20Hz).
  • Processing: The authors used 3D bounding box labels for vehicles, interpolated to match the higher-frequency Lidar frames, providing a ground truth for evaluating their "timely" fusion results.

Key Insights and Ideas

  • Temporal Redundancy is Key: The "aha moment" here is that deep learning models like MVDNet already incorporate historical data. This inherent redundancy means the model can handle a bit of lag if it's trained to expect it.
  • Don't Wait, Just Compensate: The paper proves that "perfect synchronization" is a luxury, not a necessity. By teaching the model to understand the relationship between a current Lidar frame and a slightly older Radar frame, you can quintuple the output frequency (from 4Hz to 20Hz) without needing faster hardware.
  • Shared Features, Unique Fusion: The authors found that the "low-level" features (edges, shapes) are the same regardless of timing. Only the "fusion" layers need to be adjusted to handle the temporal offset. This allowed them to create a memory-efficient multi-branch model.

Key Results

  • Frequency Boost: The system successfully increased the fusion frequency from 4Hz to 20Hz, matching the Lidar's peak performance.
  • Accuracy Recovery: Without the proposed training, using unaligned data caused Average Precision (AP) to drop from 0.735 to 0.486 (at IoU 0.8). With their Separate Training strategy, they recovered nearly all of it, reaching an AP of 0.722.
  • Efficiency: Using the Frame Skipping method, they reduced the number of historical frames required from 4 to 2 while maintaining comparable accuracy, significantly lowering the latency of the detection pipeline.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Dynamic Complexity: The current study focuses on vehicle detection. Future work could examine how these temporal offsets affect smaller, faster-moving objects like cyclists or pedestrians, where a 50ms lag might be more critical.
  • Hardware Diversity: The method was tested on specific 4Hz/20Hz hardware. Testing on a wider variety of sensor suites (like 10Hz Radar) would prove the universal applicability of the multi-branch approach.
  • Real-world Compute: While frame skipping helps, the multi-branch model adds some architectural complexity that needs to be optimized for low-power embedded chips found in actual vehicles.

Summary updated

Abstract

Fusing Radar and Lidar sensor data can fully utilize their complementary advantages and provide more accurate reconstruction of the surrounding for autonomous driving systems. Surround Radar/Lidar can provide 360° view sampling with the minimal cost, which are promising sensing hardware solutions for autonomous driving systems. However, due to the intrinsic physical constraints, the rotating speed of surround Radar, and thus the frequency to generate Radar data frames, is much lower than surround Lidar. Existing Radar/Lidar fusion methods have to work at the low frequency of surround Radar, which cannot meet the high responsiveness requirement of autonomous driving systems. This paper develops techniques to fuse surround Radar/Lidar with working frequency only limited by the faster surround Lidar instead of the slower surround Radar, based on widely-used object detection model called MVDNet. The basic idea of our approach is simple: we let MVDNet work with temporally unaligned data from Radar/Lidar, so that fusion can take place at any time when a new Lidar data frame arrives, instead of waiting for the slow Radar data frame. However, directly applying MVDNet to temporally unaligned Radar/Lidar data greatly degrades its object detection accuracy. The key information revealed in this paper is that we can achieve high output frequency with little accuracy loss by enhancing the training procedure to explore the temporal redundancy in MVDNet so that it can tolerate the temporal unalignment of input data. We explore several different ways of training enhancement and compare them quantitatively with experiments.
BibTeX
@inproceedings{xie2024timely,
  author = {Wenjing Xie and Tao Hu and Neiwen Ling and Guoliang Xing and Chun Jason Xue and Nan Guan},
  title = {{Timely Fusion of Surround Radar/Lidar for Object Detection in Autonomous Driving Systems}},
  booktitle = {RTCSA},
  year = {2024},
  doi = {10.1109/rtcsa62462.2024.00014},
}

Loading related content...

Figure