LiDAR-Based End-to-End Temporal Perception for Vehicle-Infrastructure Cooperation

Zhenwei Yang, Jilei Mao, Wenxian Yang, Yibo Ai, Yu Kong, Haibao Yu, Weidong Zhang

TL;DR

The paper introduces LET-VIC, the first LiDAR-based end-to-end tracking framework designed for Vehicle-Infrastructure Cooperation (VIC). By combining spatial data from both vehicles and roadside units with temporal historical context, and introducing a novel calibration error compensation module, the system significantly improves 3D object detection and tracking, even in the presence of sensor misalignment and communication delays.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) often suffer from "perceptual blindness" caused by physical occlusions (like a large truck blocking the view) or the limited range of onboard sensors. Vehicle-Infrastructure Cooperation (VIC) solves this by allowing vehicles to "see" through the eyes of roadside infrastructure.

However, current VIC methods face two major hurdles:

  1. Temporal Neglect: Most existing systems treat detection as a frame-by-frame task, failing to leverage the rich sequential information that helps predict object movement and maintain identity.
  2. Calibration Errors: In the real world, the coordinate systems of a moving vehicle and a fixed roadside sensor rarely align perfectly. Small errors in calibration lead to "ghosting" or misaligned features, which degrade perception accuracy.

Methods

LET-VIC tackles these issues through a unified, end-to-end Transformer-based architecture. Its core components include:

  • BEV Feature Fusion: The system converts raw LiDAR point clouds from both the vehicle and infrastructure into a shared Bird’s Eye View (BEV) representation using a PointPillars backbone.
  • VIC Cross-Attention: This module dynamically fuses infrastructure features into the vehicle’s perspective, filling in blind spots.
  • Temporal Self-Attention: Instead of processing single frames, the model integrates historical BEV features to resolve ambiguities caused by temporary occlusions.
  • Calibration Error Compensation (CEC): This is a standout feature. It’s a self-supervised module that learns to predict and correct spatial offsets ($\Delta x, \Delta y$) between the vehicle and infrastructure sensors during the coordinate transformation process.
  • End-to-End Tracking: Inspired by TrackFormer, the model uses "detect queries" for new objects and "track queries" to follow existing ones across frames, optimizing detection and tracking simultaneously rather than as separate post-processing steps.

Datasets

The authors evaluated LET-VIC on two primary benchmarks:

  1. V2X-Seq-SPD: A large-scale, real-world sequential dataset for VIC, featuring over 15,000 pairs of cooperative frames captured at 10Hz across 95 complex urban scenes.
  2. V2X-Sim: A simulation dataset built in CARLA, used to validate the model’s generalizability across different sensor configurations and lower frame rates (5Hz).

Key Insights and Ideas

  • The Synergy of Space and Time: The biggest "aha moment" is that spatial cooperation (VIC) and temporal modeling are mutually reinforcing. Temporal data helps "smooth out" momentary sensor noise, while infrastructure data provides the spatial context that a single vehicle simply cannot capture.
  • Self-Correction is Crucial: The CEC module proves that we don't need perfect manual calibration to achieve high accuracy. By making the alignment error a learnable parameter, the system becomes robust to the messy realities of hardware deployment.
  • End-to-End Beats "Detect-then-Track": By unifying the pipeline, the model avoids the compounding errors often seen in traditional tracking-by-detection methods, where a single missed detection can break an entire track.

Key Results

LET-VIC demonstrated substantial performance gains over state-of-the-art baselines:

  • Vs. Vehicle-Only (LET-V): Achieved a +15.0% improvement in mAP (Mean Average Precision) and a +17.3% boost in AMOTA (Average Multi-Object Tracking Accuracy).
  • Vs. Traditional VIC Models: Outperformed representative models like V2VNet and FFNet by at least +13.7% in mAP and +13.1% in AMOTA.
  • Robustness to Latency: The model maintained strong performance even with communication delays up to 300ms, a critical requirement for real-world V2X (Vehicle-to-Everything) applications where network jitter is common.

Limitations and Future Work

While highly effective, the authors note a few areas for growth:

  • LiDAR-Only Constraints: The system relies solely on point clouds. While LiDAR provides excellent depth, it lacks the rich semantic/color information of cameras, which can be vital for fine-grained attribute recognition.
  • Point Sparsity: At long distances or in complex environments, LiDAR points can become sparse, making it harder to identify small objects.
  • Future Directions: The team suggests exploring multi-modal fusion (LiDAR + Camera) and developing more advanced strategies to explicitly mitigate the impact of extreme communication delays or packet loss in V2X networks.

Summary updated

Abstract

Temporal perception, defined as the capability to detect and track objects across temporal sequences, serves as a fundamental component in autonomous driving systems. While single-vehicle perception systems encounter limitations, stemming from incomplete perception due to object occlusion and inherent blind spots, cooperative perception systems present their own challenges in terms of sensor calibration precision and positioning accuracy. To address these issues, we introduce LET-VIC, a LiDAR-based End-to-End Tracking framework for Vehicle-Infrastructure Cooperation (VIC). First, we employ Temporal Self-Attention and VIC Cross-Attention modules to effectively integrate temporal and spatial information from both vehicle and infrastructure perspectives. Then, we develop a novel Calibration Error Compensation (CEC) module to mitigate sensor misalignment issues and facilitate accurate feature alignment. Experiments on the V2X-Seq-SPD dataset demonstrate that LET-VIC significantly outperforms baseline models. Compared to LET-V, LET-VIC achieves +15.0% improvement in mAP and a +17.3% improvement in AMOTA. Furthermore, LET-VIC surpasses representative Tracking by Detection models, including V2VNet, FFNet, and PointPillars, with at least a +13.7% improvement in mAP and a +13.1% improvement in AMOTA without considering communication delays, showcasing its robust detection and tracking performance. The experiments demonstrate that the integration of multi-view perspectives, temporal sequences, or CEC in end-to-end training significantly improves both detection and tracking performance. All code will be open-sourced.
BibTeX
@article{yang2025lidar,
  author = {Zhenwei Yang and Jilei Mao and Wenxian Yang and Yibo Ai and Yu Kong and Haibao Yu and Weidong Zhang},
  title = {{LiDAR-Based End-to-End Temporal Perception for Vehicle-Infrastructure Cooperation}},
  journal = {IEEE Internet of Things Journal},
  year = {2025},
  doi = {10.1109/jiot.2025.3552526},
}

Loading related content...

Figure