Multitarget-Tracking Method Based on the Fusion of Millimeter-Wave Radar and LiDAR Sensor Information for Autonomous Vehicles

Junren Shi, Yingjie Tang, Jun Gao, Changhao Piao, Zhongquan WangSensors 2023 · DOI

TL;DR

This paper presents a novel multitarget tracking method for autonomous vehicles by fusing data from millimeter-wave radar and LiDAR. The approach addresses limitations in current methods by integrating sensor registration, track association, and data fusion within a distributed framework, leading to significantly improved tracking accuracy and reliability in complex, high-speed driving scenarios.

What Problem Does This Paper Address?

Intelligent driving systems rely heavily on accurate environmental perception, with multitarget tracking being a critical component. While millimeter-wave radar and LiDAR are standard sensors for this task, existing fusion methods often struggle with data accuracy and reliability, especially in complex scenarios with multiple moving targets. These challenges stem from issues like sensor registration (aligning data from different sensors in space and time), track association (determining which detections belong to the same object across sensors), and the inherent limitations of individual sensors (e.g., radar's lack of detailed geometric information, LiDAR's sparsity at long distances). This paper aims to overcome these limitations by proposing a comprehensive multitarget tracking framework that robustly integrates these key technologies.

Methods

The proposed method is built upon a Distributed Multisensor Multitarget Tracking (DMMT) system. The core contributions and steps include:

  • Single-Sensor Target Tracking: Specialized methods are developed for both millimeter-wave radar (using an extended target tracker based on the GM-PHD algorithm and a rectangular target model) and LiDAR (using a JPDA tracker with an IMM-UKF algorithm and a bounding box detector). This ensures robust initial tracking from each sensor.
  • Sensor Spatiotemporal Registration:
    • Temporal Registration: An IMM-TF (Interacting Multiple Model - Temporal Filter) method, based on Kalman filtering, is employed to handle asynchronous measurements and estimate relative time offsets between sensors.
  • Spatial Registration: The Residual Bias Estimation Registration (RBER) method is used to estimate and compensate for system deviations and ensure accurate spatial alignment of sensor data.
  • Sensor Track Association: A sequential m-best track association algorithm based on new target density (SMBTANTD) is utilized. This method iteratively correlates tracks from different sensors, improving robustness by considering historical track information.
  • Heterogeneous Sensor Fusion: An IF (Information Filter) heterogeneous sensor fusion algorithm is applied to optimally combine the track information from millimeter-wave radar and LiDAR. This approach aims to leverage the strengths of each sensor while mitigating their weaknesses, resulting in a stable and high-precision global track.

Datasets

The paper describes a simulation-based verification. The simulation scene involves an autonomous vehicle and four target vehicles in a high-speed driving scenario, including car-following and overtaking maneuvers. Specific parameters for the vehicles (speed, dimensions) and sensors (mounting location, field of view, resolution) are provided in Tables 1 and 2. No real-world datasets were used for training or evaluation.

Key Insights and Ideas

  • Synergistic Fusion is Crucial: The paper strongly advocates that simply combining radar and LiDAR data isn't enough. Robust spatiotemporal registration and sophisticated track association are essential prerequisites for effective fusion.
  • Addressing Sensor Limitations: The method explicitly tackles the weaknesses of individual sensors. For instance, the extended target tracking for radar helps with its inherent limitations in geometric detail, while the bounding box detection for LiDAR addresses its sparsity.
  • Distributed Architecture Advantage: The choice of a DMMT framework simplifies implementation and improves scalability compared to centralized approaches, while still achieving high performance.
  • Complex Scenarios are the Real Test: The focus on high-speed driving with multiple maneuvering targets highlights the practical challenges and the need for advanced tracking solutions beyond simple scenarios.

Key Results

The proposed multitarget-tracking method demonstrated significant improvements over single-sensor trackers:

  • Reduced Estimation Errors:
    • Position error reduced by 85.5%.
  • Velocity error reduced by 64.6%.
  • Size error reduced by 75.3%.
  • Direction error reduced by 9.5%. (Note: Size and direction errors are not shown for the radar-only tracker due to its limitations.)
  • Improved GOSPA Metric: The average GOSPA (Generalized Optimal SubPattern Assignment) indicator was reduced by 19.8%, indicating better overall tracking accuracy.
  • Enhanced Robustness: The fusion tracker effectively handles scenarios where targets are very close, preventing the lidar tracker from producing target points that deviate from the vehicle's center due to aggregated point clouds.

Limitations and Future Work

  • Simulation-Based: The evaluation was conducted solely through simulation. While the simulation scene is detailed, real-world testing is necessary to fully validate the method's performance under diverse environmental conditions.
  • Future Work: The authors suggest extending the research to more challenging scenarios, including adverse weather conditions (rainy, snowy days), complex environments like viaducts and tunnels, and exploring different sensor combinations (e.g., adding cameras). They also plan to conduct real-vehicle experiments to further verify the proposed method.

Summary updated

Abstract

Multitarget tracking based on multisensor fusion perception is one of the key technologies to realize the intelligent driving of automobiles and has become a research hotspot in the field of intelligent driving. However, most current autonomous-vehicle target-tracking methods based on the fusion of millimeter-wave radar and lidar information struggle to guarantee accuracy and reliability in the measured data, and cannot effectively solve the multitarget-tracking problem in complex scenes. In view of this, based on the distributed multisensor multitarget tracking (DMMT) system, this paper proposes a multitarget-tracking method for autonomous vehicles that comprehensively considers key technologies such as target tracking, sensor registration, track association, and data fusion based on millimeter-wave radar and lidar. First, a single-sensor multitarget-tracking method suitable for millimeter-wave radar and lidar is proposed to form the respective target tracks; second, the Kalman filter temporal registration method and the residual bias estimation spatial registration method are used to realize the temporal and spatial registration of millimeter-wave radar and lidar data; third, use the sequential m-best method based on the new target density to find the track the correlation of different sensors; and finally, the IF heterogeneous sensor fusion algorithm is used to optimally combine the track information provided by millimeter-wave radar and lidar, and finally form a stable and high-precision global track. In order to verify the proposed method, a multitarget-tracking simulation verification in a high-speed scene is carried out. The results show that the multitarget-tracking method proposed in this paper can realize the track tracking of multiple target vehicles in high-speed driving scenarios. Compared with a single-radar tracker, the position, velocity, size, and direction estimation errors of the track fusion tracker are reduced by 85.5%, 64.6%, 75.3%, and 9.5% respectively, and the average value of GOSPA indicators is reduced by 19.8%; more accurate target state information can be obtained than a single-radar tracker.
BibTeX
@inproceedings{shi2023multitarget,
  author = {Junren Shi and Yingjie Tang and Jun Gao and Changhao Piao and Zhongquan Wang},
  title = {{Multitarget-Tracking Method Based on the Fusion of Millimeter-Wave Radar and LiDAR Sensor Information for Autonomous Vehicles}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23156920},
}
Figure