LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion

Meet Shah, Zhiling Huang, Ankit Laddha, Matthew Langford, Blake Barber, Sidney Zhang, Carlos Vallespi-Gonzalez, Raquel Urtasun

Abstract

In this paper, we present LiRaNet, a novel end-to-end trajectory prediction method which utilizes radar sensor information along with widely used lidar and high definition (HD) maps. Automotive radar provides rich, complementary information, allowing for longer range vehicle detection as well as instantaneous radial velocity measurements. However, there are factors that make the fusion of lidar and radar information challenging, such as the relatively low angular resolution of radar measurements, their sparsity and the lack of exact time synchronization with lidar. To overcome these challenges, we propose an efficient spatio-temporal radar feature extraction scheme which achieves state-of-the-art performance on multiple large-scale datasets. Further, by incorporating radar information, we show a 52% reduction in prediction error for objects with high acceleration and a 16% reduction in prediction error for objects at longer range.

Introduction

Accurate motion prediction of surrounding actors is a cornerstone of safe autonomous driving. While lidar-based systems have become the standard due to their high-resolution 3D point clouds, they suffer from inherent limitations. Lidar primarily provides positional data, requiring multiple frames to estimate velocity and acceleration, which introduces dangerous latency in dynamic scenarios. Furthermore, lidar point density degrades significantly at long ranges and under occlusion.

Radar offers a compelling solution by providing instantaneous radial velocity via the Doppler effect and maintaining performance at greater distances. However, radar data is notoriously noisy and sparse, with poor angular resolution compared to lidar. LiRaNet addresses these challenges by effectively fusing radar, lidar, and HD maps in an end-to-end framework.

Methodology

The core of LiRaNet is a spatio-temporal feature learning scheme designed to handle the irregular nature of radar data.

Spatio-Temporal Radar Feature Learning

Unlike lidar points which can be easily projected onto a grid, radar points are sparse and unordered. LiRaNet employs parametric continuous convolutions to map these points onto a Bird’s Eye View (BEV) grid. By processing a temporal sequence of radar sweeps, the model can recover the full 2D velocity and yaw rate of actors, effectively bypassing the latency associated with lidar-only dynamics estimation.

Multi-Sensor Fusion

The architecture integrates features from three distinct sources: radar, lidar (voxelized occupancy grids), and HD maps (rasterized layers). These features are concatenated and processed through a multi-scale fusion backbone. The final prediction head outputs multiple probabilistic trajectories to account for multi-modal future behaviors.

LiRaNet architecture showing (A) radar feature extraction, (B) the fusion backbone, and (C) the final prediction output.

Experimental Results

LiRaNet was evaluated on the public nuScenes dataset and Uber ATG’s large-scale X17k dataset. The inclusion of radar led to substantial performance gains, particularly in "edge case" scenarios where lidar is most vulnerable.

  • High Dynamics: For objects undergoing rapid acceleration or deceleration, LiRaNet achieved a 52% reduction in prediction error. Radar's ability to sense speed changes instantly provides a "dynamics shortcut" that lidar-only models lack.
  • Long-Range Performance: At distances between 70m and 100m, where lidar points are sparse, radar fusion resulted in a 16% error reduction.
  • Real-Time Viability: The system maintains a low inference latency of approximately 40ms, making it suitable for real-world self-driving stacks.
Comparison of trajectory prediction with and without radar. The red arrows represent radar returns providing instantaneous velocity cues that stabilize the predicted path.

Conclusion

LiRaNet demonstrates that radar is a powerful, yet often underutilized, sensor for motion prediction. By overcoming the challenges of radar noise and sparsity through spatio-temporal feature learning, the model significantly improves safety in high-acceleration and long-range scenarios. Future work will explore the use of raw radar tensors (Range-Azimuth-Doppler) to further enhance feature extraction and prediction accuracy.

Summary updated

Abstract

In this paper, we present LiRaNet, a novel end-to-end trajectory prediction method which utilizes radar sensor information along with widely used lidar and high definition (HD) maps. Automotive radar provides rich, complementary information, allowing for longer range vehicle detection as well as instantaneous radial velocity measurements. However, there are factors that make the fusion of lidar and radar information challenging, such as the relatively low angular resolution of radar measurements, their sparsity and the lack of exact time synchronization with lidar. To overcome these challenges, we propose an efficient spatio-temporal radar feature extraction scheme which achieves state-of-the-art performance on multiple large-scale datasets.Further, by incorporating radar information, we show a 52% reduction in prediction error for objects with high acceleration and a 16% reduction in prediction error for objects at longer range.
BibTeX
@inproceedings{shah2020liranet,
  author = {Meet Shah and Zhiling Huang and Ankit Laddha and Matthew Langford and Blake Barber and Sidney Zhang and Carlos Vallespi-Gonzalez and Raquel Urtasun},
  title = {{LiRaNet: End-to-End Trajectory Prediction using Spatio-Temporal Radar Fusion}},
  booktitle = {Conference on Robot Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=21PnM6kpcn},
}
Figure