CoFormerNet: A Transformer-Based Fusion Approach for Enhanced Vehicle-Infrastructure Cooperative Perception

Bin Li, Yanan Zhao, Huachun Tan2024 · DOI

TL;DR

This paper introduces CoFormerNet, a novel transformer-based framework for vehicle-infrastructure cooperative 3D object detection. It tackles the challenge of temporal and spatial misalignments between vehicle and infrastructure sensors, achieving state-of-the-art results on benchmark datasets by effectively fusing historical infrastructure data with real-time vehicle data.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on accurate 3D object detection to navigate safely. While onboard sensors like LiDAR are crucial, they have limitations such as blind spots and limited range. Cooperative perception, which leverages infrastructure-side sensors (e.g., roadside units), aims to overcome these limitations by providing a broader perspective and longer detection range. However, a significant challenge in vehicle-infrastructure cooperative perception is the temporal and spatial misalignment between data from vehicle-mounted and roadside sensors. This misalignment, caused by communication delays and differences in sensor placement, can lead to inaccurate object detection and reduced system performance. Existing methods often struggle to effectively fuse this asynchronous and spatially offset data, especially in complex traffic scenarios like intersections.

Methods

CoFormerNet proposes a unified, end-to-end trainable transformer-based framework for vehicle-infrastructure cooperative 3D object detection. The core innovations lie in how it handles temporal and spatial discrepancies:

  • Temporal Aggregation Module (TAM): For the infrastructure branch, TAM fuses historical Bird's Eye View (BEV) features. Crucially, for the vehicle branch, TAM treats the infrastructure's temporally fused BEV features as historical data, effectively mitigating the impact of communication delays without explicit delay prediction.
  • Spatial-Modulated Cross-Attention (SMCA): To address spatial misalignment between vehicle and infrastructure LiDAR data, CoFormerNet employs an SMCA module. This module establishes a soft association between vehicle and infrastructure features, allowing the network to adaptively focus on relevant spatial information and reducing reliance on precise sensor calibration.
  • Unified Positional Encoding: To ensure consistency across disparate spatial domains, positional encodings from the infrastructure side are transformed to align with the vehicle's coordinate system.
  • Feature Transformation, Compression, and Decompression: To manage communication bandwidth and latency, infrastructure-side features are transformed into the vehicle's coordinate system, compressed, transmitted, and then decompressed by the vehicle.
  • End-to-End Training: The entire framework is trained end-to-end, allowing for better optimization of the fusion process and improved overall performance compared to staged training approaches.

Datasets

The paper evaluates CoFormerNet on two datasets:

  • DAIR-V2X: This dataset captures real-world urban driving scenarios, including highways and intersections, with synchronized vehicle and infrastructure LiDAR data. It is highlighted as a more realistic benchmark than purely simulated data.
  • V2XSet: This dataset is generated using simulations (CARLA and OpenCDA) and includes introduced time delays, making it suitable for evaluating robustness to latency.

Key Insights and Ideas

  • Temporal Fusion as a Solution to Delay: Instead of trying to predict or compensate for communication delays directly, CoFormerNet cleverly reframes the problem as integrating historical infrastructure data. This is a more robust approach to handling inherent uncertainties.
  • Soft Association for Spatial Misalignment: The SMCA module is key to overcoming the challenges posed by imperfect sensor calibration and spatial differences. It allows the network to learn where to look, rather than relying on rigid associations.
  • Unified Framework Benefits: An end-to-end trainable, unified model simplifies the training process and leads to better performance by allowing all components to learn cooperatively.
  • Infrastructure as a "Memory Bank": The infrastructure's fixed position and ability to store historical data make it an invaluable "memory bank" for the vehicle, providing context that the vehicle's sensors alone cannot capture.

Key Results

CoFormerNet achieved state-of-the-art performance on the DAIR-V2X validation set, demonstrating significant improvements over existing methods across various metrics, including mAP@3D and mAP@BEV. For instance, on DAIR-V2X with 0ms latency, CoFormerNet achieved 61.03 mAP@3D and 54.59 mAP@BEV (IoU=0.7), outperforming previous bests. The model also showed remarkable robustness to latency, with minimal performance degradation at 200ms and 300ms compared to other methods. On the V2XSet dataset, CoFormerNet also outperformed other intermediate fusion methods, especially under latency conditions. Ablation studies confirmed the significant contributions of TAM, SMCA, and end-to-end training to the overall performance.

Limitations and Future Work

The paper acknowledges that while CoFormerNet achieves high accuracy, its inference time is longer than some baseline methods. The authors suggest that advancements in computational hardware are expected to bridge this gap, making CoFormerNet a viable option for real-time applications. Future work could focus on further optimizing the model for even greater computational efficiency while maintaining its high accuracy. The authors also implicitly suggest that extending this framework to incorporate other sensor modalities (like cameras) could further enhance perception capabilities.

Summary updated

BibTeX
@misc{li2024coformernet,
  author = {Bin Li and Yanan Zhao and Huachun Tan},
  title = {{CoFormerNet: A Transformer-Based Fusion Approach for Enhanced Vehicle-Infrastructure Cooperative Perception}},
  year = {2024},
  doi = {10.3390/s24134101},
}
Figure