CoInfra: A Large-Scale Cooperative Infrastructure Perception System and Dataset in Adverse Weather

Minghao Ning, Yufeng Yang, Keqi Shu, Shucheng Huang, Jiaming Zhong, Maryam Salehi, Mahdi Rahmani, Yukun Lu, Chen Sun, Aladdin Saleh, Ehsan Hashemi, Amir KhajepourarXiv 2025 · OpenReview

TL;DR

This paper introduces CoInfra, a comprehensive system and dataset for vehicle-infrastructure (V2I) cooperative perception, designed to overcome the limitations of onboard-only sensing in adverse weather and complex urban environments. CoInfra demonstrates that by fusing data from multiple roadside sensors, situational awareness can be significantly improved, especially in safety-critical conflict scenarios.

What Problem Does This Paper Address?

Autonomous driving systems rely heavily on perception to understand their surroundings. However, current systems struggle with occlusions, limited sensor range, and adverse weather conditions (like rain, snow, or fog), which severely degrade sensor performance. While vehicle-to-infrastructure (V2I) cooperation offers a promising solution by augmenting vehicle sensors with roadside infrastructure, existing research and datasets have limitations: they often lack large-scale, real-world deployments, realistic communication constraints (like 5G latency), and diverse weather conditions. Quantifying the actual safety benefits of V2I beyond standard perception metrics has also been a challenge. This paper aims to bridge these gaps by providing a deployable V2I system, a comprehensive dataset, and a new evaluation methodology focused on transportation-facing safety.

Methods

The CoInfra system is built around 14 roadside sensor nodes, each equipped with LiDAR, cameras, and edge computing, all connected via a commercial 5G network. Key methodological contributions include:

  • Delay-Aware Synchronization: A novel approach to synchronize sensor data from multiple nodes despite stochastic 5G communication delays. This involves online estimation of latency and an adaptive fusion window to balance completeness and responsiveness.
  • Full-Stack System: CoInfra provides an end-to-end system, from hardware design and calibration to a ROS-based perception pipeline and a management UI, enabling researchers to replicate and extend the system.
  • Multi-Node Perception Pipeline: Each infrastructure node performs local perception (object detection and tracking) on edge devices, transmitting only structured outputs to a central cloud for fusion. This is crucial for managing 5G bandwidth and latency.
  • Transportation-Facing Evaluation: Beyond standard detection metrics (like Average Precision), the paper introduces an "ego-centric observability" metric to quantify how V2I cooperation improves awareness of safety-critical participants from the perspective of a connected vehicle, particularly in conflict scenarios.

Datasets

The paper introduces the CoInfra dataset, comprising two subsets:

  • CoInfra-I2I: The primary dataset, featuring synchronized observations from eight infrastructure sensor nodes deployed around an urban roundabout. It covers four weather conditions (sunny, rainy, heavy snow, freezing rain) and includes 294k LiDAR frames, 589k camera images, and 332k globally consistent 3D bounding boxes. This subset is designed for large-scale benchmarking of cooperative perception.
  • CoInfra-V2I: A smaller subset collected during coordinated operation between the infrastructure nodes and an autonomous shuttle bus. This subset includes synchronized data from both infrastructure and vehicle sensors, enabling system-level validation and V2I experiments.

Key Insights and Ideas

  • 5G Communication Trade-offs: Transmitting raw sensor data (like images) over 5G is impractical due to high latency and packet loss. Lightweight, object-level outputs are essential for real-time V2I cooperation.
  • Adaptive Synchronization is Crucial: Simply waiting for all sensors to report (wait-for-all) leads to high reaction times. An adaptive synchronization strategy, combined with lightweight state prediction (EKF), significantly reduces tail latency while maintaining high detection completeness.
  • Global Fusion Dominates Local Fusion: For object detection, especially for sparse classes like VRUs (Vulnerable Road Users) and at longer distances, fusing data globally (early fusion) yields significantly better performance than local fusion (late fusion).
  • Adverse Weather Impacts VRUs More: While adverse weather degrades perception for all object types, it disproportionately affects VRU detection, highlighting the need for robust cooperative sensing in these conditions.
  • Situational Awareness is the True Value: The most compelling benefit of V2I cooperation isn't just higher detection scores, but a dramatic improvement in "critical-frame completeness" – ensuring that all safety-critical participants are observable when needed most, particularly in complex roundabout interactions where single-vehicle sensing is severely limited.

Key Results

  • Communication Efficiency: Object-level detections (1-2 KB) have a mean latency of 24.2 ms and P95 of 38.2 ms with zero packet loss. Compressed images (500-600 KB) suffer from over 2.19 s mean latency and 63.75% packet drop.
  • Synchronization Performance: Adaptive synchronization with EKF reduces P99 reaction time from 75.7 ms to 57.0 ms while maintaining full fusion completeness.
  • Detection Benchmarks: Global fusion models achieve significantly higher [email protected] than local fusion, especially for VRUs (e.g., 0.720 vs. 0.320 for GlobalLidar vs. LocalLidar in VRU crossing zones). Adverse weather degrades VRU AP from 0.654 (sunny) to 0.585 (snow) for GlobalLidarCam.
  • Ego-Centric Observability: In critical roundabout scenarios, vehicle-only critical-frame completeness for VRU waiting increases from 33.0% to 85.9% with V2I. For vehicles in conflict, it rises from 46.1% to 100.0%. Overall, V2I sensing restores observability in scenarios where vehicle-only sensing is most limited.

Limitations and Future Work

  • CoInfra-V2I Subset Size: The V2I subset is smaller and currently limited to clear-weather runs, making it more of a validation tool than a comprehensive benchmark for V2I.
  • Observability Proxy: The ego-centric observability analysis uses LiDAR point support as a proxy for situational awareness, which doesn't fully replace closed-loop assessment with planning and control.
  • Operating Point Mismatch: The strongest global fusion benchmarks don't perfectly match the deployed real-time system's compact object-level operating mode. Future work will investigate intermediate operating points like real-time point-cloud fusion.
  • Future Work: The authors plan to expand the V2I subset to include more diverse weather conditions and vehicle types, investigate practical intermediate fusion strategies, and extend the evaluation to cooperative tracking, motion prediction, and planning-level closed-loop assessment.

Summary updated

BibTeX
@misc{ning2025coinfra,
  author = {Minghao Ning and Yufeng Yang and Keqi Shu and Shucheng Huang and Jiaming Zhong and Maryam Salehi and Mahdi Rahmani and Yukun Lu and Chen Sun and Aladdin Saleh and Ehsan Hashemi and Amir Khajepour},
  title = {{CoInfra: A Large-Scale Cooperative Infrastructure Perception System and Dataset in Adverse Weather}},
  year = {2025},
  url = {https://openreview.net/forum?id=yjJFKtG5xd},
}
Figure