LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation

Zhenwei Yang, Jilei Mao, Wenxian Yang, Yibo Ai, Yu Kong, Haibao Yu, Weidong Zhang

Abstract

Temporal perception, defined as the capability to detect and track objects across temporal sequences, serves as a fundamental component in autonomous driving systems. While single-vehicle perception systems encounter limitations, stemming from incomplete perception due to object occlusion and inherent blind spots, cooperative perception systems present their own challenges in terms of sensor calibration precision and positioning accuracy. To address these issues, we introduce LET-VIC, a LiDAR-based End-to-End Tracking framework for Vehicle-Infrastructure Cooperation (VIC). First, we employ Temporal Self-Attention and VIC Cross-Attention modules to effectively integrate temporal and spatial information from both vehicle and infrastructure perspectives. Then, we develop a novel Calibration Error Compensation (CEC) module to mitigate sensor misalignment issues and facilitate accurate feature alignment. Experiments on the V2X-Seq-SPD dataset demonstrate that LET-VIC significantly outperforms baseline models. Compared to LET-V, LET-VIC achieves +15.0% improvement in mAP and a +17.3% improvement in AMOTA. Furthermore, LET-VIC surpasses representative Tracking by Detection models, including V2VNet, FFNet, and PointPillars, with at least a +13.7% improvement in mAP and a +13.1% improvement in AMOTA without considering communication delays, showcasing its robust detection and tracking performance. All code will be open-sourced.

Introduction

Autonomous vehicles (AVs) often suffer from "perceptual blindness" caused by physical occlusions (like a large truck blocking the view) or the limited range of onboard sensors. Vehicle-Infrastructure Cooperation (VIC) solves this by allowing vehicles to "see" through the eyes of roadside infrastructure.

Comparison of infrastructure-side (a), vehicle-side (b), and cooperative (c) perception views. Cooperation enables the detection of occluded and out-of-view objects.

However, current VIC methods face two major hurdles: Temporal Neglect, where systems fail to leverage sequential information for movement prediction, and Calibration Errors, where misaligned coordinate systems between moving vehicles and fixed roadside sensors lead to degraded accuracy. LET-VIC addresses these by unifying spatial cooperation and temporal modeling in an end-to-end framework.

Methodology

LET-VIC utilizes a unified, end-to-end Transformer-based architecture to process multi-view temporal sequences.

The LET-VIC architecture, featuring infrastructure and vehicle backbones, VIC Cross-Attention, Temporal Self-Attention, and the Calibration Error Compensation (CEC) module.

Core Components

  • BEV Feature Fusion: The system converts raw LiDAR point clouds from both the vehicle and infrastructure into a shared Bird’s Eye View (BEV) representation using a PointPillars backbone.
  • VIC Cross-Attention: This module dynamically fuses infrastructure features into the vehicle’s perspective, filling in blind spots and extending the perception range.
  • Temporal Self-Attention: Instead of processing single frames, the model integrates historical BEV features to resolve ambiguities caused by temporary occlusions and sensor noise.
  • Calibration Error Compensation (CEC): A standout self-supervised module that learns to predict and correct spatial offsets ($\Delta x, \Delta y$) between the vehicle and infrastructure sensors during the coordinate transformation process, ensuring robust feature alignment.
  • End-to-End Tracking: Inspired by TrackFormer, the model uses "detect queries" for new objects and "track queries" to follow existing ones across frames, optimizing detection and tracking simultaneously.

Experimental Results

The authors evaluated LET-VIC on the V2X-Seq-SPD real-world sequential dataset and the V2X-Sim simulation benchmark.

Key performance highlights include:

  • Superior Accuracy: Compared to a vehicle-only baseline (LET-V), LET-VIC achieved a +15.0% improvement in mAP and a +17.3% boost in AMOTA. It also outperformed representative VIC models like V2VNet and FFNet by over 13% in both metrics.
  • Latency Robustness: The model maintained strong performance even with communication delays up to 300ms, a critical requirement for real-world V2X applications where network jitter is common.
  • Synergy of Space and Time: Results indicate that spatial cooperation and temporal modeling are mutually reinforcing; temporal data smooths momentary noise, while infrastructure data provides essential spatial context.

Conclusion and Future Work

LET-VIC demonstrates that integrating multi-view perspectives, temporal sequences, and learnable calibration compensation significantly improves both detection and tracking performance in cooperative systems. While highly effective, the current LiDAR-only approach faces challenges with point sparsity at long distances. Future research will explore multi-modal fusion (LiDAR + Camera) to incorporate rich semantic information and develop advanced strategies to mitigate extreme communication delays or packet loss in V2X networks.

Summary updated

Abstract

Temporal perception, defined as the capability to detect and track objects across temporal sequences, serves as a fundamental component in autonomous driving systems. While single-vehicle perception systems encounter limitations, stemming from incomplete perception due to object occlusion and inherent blind spots, cooperative perception systems present their own challenges in terms of sensor calibration precision and positioning accuracy. To address these issues, we introduce LET-VIC, a LiDAR-based End-to-End Tracking framework for Vehicle-Infrastructure Cooperation (VIC). First, we employ Temporal Self-Attention and VIC Cross-Attention modules to effectively integrate temporal and spatial information from both vehicle and infrastructure perspectives. Then, we develop a novel Calibration Error Compensation (CEC) module to mitigate sensor misalignment issues and facilitate accurate feature alignment. Experiments on the V2X-Seq-SPD dataset demonstrate that LET-VIC significantly outperforms baseline models. Compared to LET-V, LET-VIC achieves +15.0% improvement in mAP and a +17.3% improvement in AMOTA. Furthermore, LET-VIC surpasses representative Tracking by Detection models, including V2VNet, FFNet, and PointPillars, with at least a +13.7% improvement in mAP and a +13.1% improvement in AMOTA without considering communication delays, showcasing its robust detection and tracking performance. The experiments demonstrate that the integration of multi-view perspectives, temporal sequences, or CEC in end-to-end training significantly improves both detection and tracking performance. All code will be open-sourced.
BibTeX
@article{yang2025lidar,
  author = {Zhenwei Yang and Jilei Mao and Wenxian Yang and Yibo Ai and Yu Kong and Haibao Yu and Weidong Zhang},
  title = {{LiDAR-based End-to-end Temporal Perception for Vehicle-Infrastructure Cooperation}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2025.3552526},
}
Figure