BEVSync: Asynchronous Data Alignment for Camera-based Vehicle-Infrastructure Cooperative Perception Under Uncertain Delays

Wentao Wang, Jiaqian Wang, Yuxin Deng, Guang Tan

TL;DR

This paper introduces BEVSync, a novel system for camera-based vehicle-infrastructure (V2I) cooperative perception that tackles the challenge of uncertain communication delays. By adaptively aligning asynchronous data from roadside cameras and ego-vehicles using an "extractor-compensator" model, BEVSync significantly improves object detection accuracy and robustness in real-world scenarios.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate environmental perception. Vehicle-to-everything (V2X) communication, particularly vehicle-to-infrastructure (V2I), offers a promising avenue to enhance this perception by allowing vehicles to exchange information with roadside infrastructure. While many V2I systems focus on LiDAR, cameras are more cost-effective and widely deployed. However, a critical challenge in camera-based V2I is the temporal asynchrony between data streams from the vehicle and infrastructure. This arises from clock differences and varying processing/transmission times, leading to uncertain delays that complicate data alignment and degrade perception accuracy. Existing methods often assume perfect synchronization or rely on simpler delay compensation, which can be insufficient in dynamic real-world conditions.

Methods

BEVSync proposes a novel asynchronous perception fusion framework. Its core innovation lies in an extractor-compensator model designed to handle uncertain delays.

  • Extractor: This module runs on the infrastructure side. It takes historical camera frames and extracts motion information. This is achieved using a deformable attention mechanism that queries historical features and their temporal encodings to capture movement patterns.
  • Compensator: This module runs on the ego-vehicle. It receives the motion-infused features from the infrastructure and uses them to predict future infrastructure features, effectively compensating for the transmission delay and aligning the data temporally. This prediction allows for a more accurate fusion with the ego-vehicle's own perception data.
  • Feature Fusion: After temporal alignment, the infrastructure and ego-vehicle features are spatially aligned (using a bird's-eye-view representation) and then aggregated for the final object detection.

The system uses a ResNet-50 backbone for feature extraction and BEVHeight for generating Bird's-Eye-View (BEV) features.

Datasets

The primary dataset used for training and evaluation is the DAIR-V2X dataset. This is a real-world dataset that provides cooperative 3D annotations for vehicle-infrastructure scenarios, including camera and LiDAR data. The authors specifically focus on the camera-based part of the dataset.

Key Insights and Ideas

  • Temporal Asynchrony is a Major Hurdle: The paper strongly emphasizes that ignoring or inadequately addressing temporal delays significantly harms cooperative perception performance, especially under longer delays.
  • Motion Prediction is Key to Compensation: Simply fusing data at the closest available timestamps is insufficient. The ability to predict future states based on historical motion is crucial for effective alignment when delays are uncertain.
  • Extractor-Compensator Synergy: The separation of motion extraction (infrastructure) and compensation (vehicle) allows for a more robust and adaptable system, as the vehicle can learn to interpret and correct for infrastructure-side delays.
  • BEV Representation Simplifies Fusion: By projecting both vehicle and infrastructure data into a common BEV space, spatial alignment becomes more straightforward, allowing the system to focus on temporal alignment.

Key Results

BEVSync demonstrates significant improvements over existing methods on the DAIR-V2X dataset, particularly under varying transmission delays:

  • Superior Performance: BEVSync consistently outperforms other methods, achieving the best results across various metrics (mAP@3D and mAP@BEV) under both 100 ms and 300 ms transmission delays.
  • Robustness to Delays: Even under a 300 ms delay, BEVSync achieves a mAP@3D of 28.33 (IoU0.3) and 13.80 (IoU0.5), outperforming FFNet (26.59 and 13.14) and Late + Kalman (18.67 and 10.12).
  • Robustness to Clock Differences: When tested with random clock differences ranging from -50 ms to 50 ms, BEVSync maintains its performance, indicating its ability to handle real-world clock inaccuracies.
  • Qualitative Improvements: Visualizations show that BEVSync provides more accurate and stable detection results compared to VIMI and FFNet, especially for dynamic targets and complex trajectories, even under significant delays.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the extractor can slightly improve performance even in synchronized systems due to capturing occluded objects from historical frames, its primary benefit is in handling asynchronous scenarios.
  • The performance of all methods degrades as transmission delays increase beyond 700 ms, though BEVSync shows a milder decline.
  • Future Work:
    • The authors suggest extending BEVSync to LiDAR-based cooperative fusion.
  • They also plan to explore heterogeneous sensor fusion to balance application cost and perception performance.

Summary updated

Abstract

Vehicle-to-infrastructure (V2I) cooperative perception systems can enhance the sensing abilities of autonomous vehicles. Existing V2I solutions often consider LiDARs devices instead of cameras, the most prevalent sensors with low cost and wide installation. In addition, a major challenge that has been underexplored is the time asynchrony between image frames from different sources. This asynchrony arises because of clock differences, varying times involved in data processing and transmission, causing uncertain delays that complicate data alignment and potentially reduce perception accuracy. We propose BEVSync, a camera-based V2I cooperative perception system that adaptively aligns frames from the ego-vehicle and infrastructure by compensating for motion deviations. Specifically, we develop an extractor-compensator model to extract and predict perceptual features using historical frames, thereby smoothing out the data misalignment. Experiments on the real-world dataset DAIR-V2X show that our approach surpasses existing methods in terms of performance and robustness.
BibTeX
@inproceedings{wang2025bevsync,
  author = {Wentao Wang and Jiaqian Wang and Yuxin Deng and Guang Tan},
  title = {{BEVSync: Asynchronous Data Alignment for Camera-based Vehicle-Infrastructure Cooperative Perception Under Uncertain Delays}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i14.33611},
}
Figure