Collaborative Perception Datasets for Autonomous Driving: A Review

N. Y. Wang, Deyong Shang, Yan Gong, Xiaoxi Hu, Ziying Song, Lei Yang, Yuhan Huang, Xiaoyu Wang, Jianli Lu

Abstract: Collaborative perception has attracted growing interest from academia and industry due to its potential to enhance perception accuracy, safety, and robustness in autonomous driving through multi-agent information fusion. With the advancement of Vehicle-to-Everything (V2X) communication, numerous collaborative perception datasets have emerged, varying in cooperation paradigms, sensor configurations, data sources, and application scenarios. However, the absence of systematic summarization and comparative analysis hinders effective resource utilization and standardization of model evaluation. As the first comprehensive review focused on collaborative perception datasets, this work reviews and compares existing resources from a multi-dimensional perspective. We categorize datasets based on cooperation paradigms, examine their data sources and scenarios, and analyze sensor modalities and supported tasks. A detailed comparative analysis is conducted across multiple dimensions. We also outline key challenges and future directions, including dataset scalability, diversity, domain adaptation, standardization, privacy, and the integration of large language models. To support ongoing research, we provide a continuously updated online repository of collaborative perception datasets and related literature: https://github.com/frankwnb/Collaborative-Perception-Datasets-for-Autonomous-Driving.

The Limitations of Single-Vehicle Perception

Single-vehicle perception systems, no matter how advanced, are fundamentally limited by their line-of-sight. Physical occlusions—such as a large truck blocking a car's view of a pedestrian—and inherent sensor range constraints create dangerous "blind spots" that lead to safety risks. Collaborative Perception (CP) addresses these risks by enabling vehicles and infrastructure to share sensor data via V2X communication. This collective intelligence provides every participating agent with a "bird’s-eye view" of the entire scene, effectively seeing through obstacles and beyond individual sensor limits.

Conceptual overview of Collaborative Perception (CP) paradigms and their role in eliminating vehicle perception blind spots.

A Multi-Dimensional Review of CP Resources

To navigate the rapidly expanding landscape of CP, this review organizes resources into five core cooperation paradigms:

  1. Roadside: Utilizing static sensors mounted on infrastructure like poles or gantries.
  2. V2V (Vehicle-to-Vehicle): Direct data sharing between moving vehicles.
  3. V2I (Vehicle-to-Infrastructure): Communication between vehicles and roadside units (RSUs).
  4. V2X (Vehicle-to-Everything): A hybrid approach integrating multiple communication channels.
  5. I2I (Infrastructure-to-Infrastructure): Linking multiple RSUs to provide continuous coverage of complex urban environments.

The Evolution of Collaborative Datasets

The field has transitioned through three distinct stages: the Formation Stage (concept proposal), the Incubation Stage (emergence of the first datasets around 2019), and the current Growth Stage, characterized by a surge in large-scale, multi-modal datasets.

Timeline and categorization of major collaborative perception datasets from 2014 to 2025.

Several "milestone" datasets have defined this trajectory:

  • Simulation Benchmarks: OPV2V and V2X-Sim, built on platforms like CARLA and SUMO, remain the gold standards for algorithm development in controlled environments.
  • Real-World Pioneers: DAIR-V2X represents the first large-scale real-world V2I dataset, while V2V4Real provides essential real-world LiDAR data for vehicle-to-vehicle collaboration.
  • Specialized Applications: DeepAccident focuses on safety-critical accident prediction, and V2X-QA explores the intersection of CP with multi-modal reasoning using Large Language Models (LLMs).
  • Infrastructure Focus: RCooper and InScope highlight the growing importance of I2I collaboration in dense urban settings.

Critical Insights and the "Sim2Real" Gap

A significant finding of this review is the quantification of the "Sim2Real" gap. Models trained on idealized synthetic data often experience a performance drop of up to 40% in Average Precision (AP) when deployed in real-world scenarios. This discrepancy is driven by real-world sensor noise, environmental variability, and communication latencies.

Furthermore, the review emphasizes that static infrastructure serves as a "God View." While V2V is beneficial, V2I and I2I provide the stable, high-vantage perspective necessary for navigating complex intersections. The emergence of datasets like V2V-QA also suggests a shift toward "cognitive" collaboration, where agents don't just share raw pixels or points but reason about the scene collectively.

Future Frontiers

Despite rapid progress, several challenges remain that current datasets haven't fully conquered:

  • Scalability & Diversity: Current datasets lack the scale of single-vehicle counterparts and are often geographically limited. More data from adverse weather (snow, heavy rain) and diverse global regions is needed.
  • Standardization: Inconsistent data formats and annotation standards hinder cross-dataset training and benchmarking.
  • Privacy & Security: Sharing high-resolution sensor data raises privacy concerns, necessitating the development of privacy-preserving CP methods.
  • Foundation Models: Leveraging Generative AI and LLMs could automate annotation and generate rare "long-tail" traffic scenarios to improve model robustness.

Summary updated

Abstract

Collaborative perception has attracted growing interest from academia and industry due to its potential to enhance perception accuracy, safety, and robustness in autonomous driving through multi-agent information fusion. With the advancement of Vehicle-to-Everything (V2X) communication, numerous collaborative perception datasets have emerged, varying in cooperation paradigms, sensor configurations, data sources, and application scenarios. However, the absence of systematic summarization and comparative analysis hinders effective resource utilization and standardization of model evaluation. As the first comprehensive review focused on collaborative perception datasets, this work reviews and compares existing resources from a multi-dimensional perspective. We categorize datasets based on cooperation paradigms, examine their data sources and scenarios, and analyze sensor modalities and supported tasks. A detailed comparative analysis is conducted across multiple dimensions. We also outline key challenges and future directions, including dataset scalability, diversity, domain adaptation, standardization, privacy, and the integration of large language models. To support ongoing research, we provide a continuously updated online repository of collaborative perception datasets and related literature: https://github.com/frankwnb/Collaborative-Perception-Datasets-for-Autonomous-Driving.
BibTeX
@misc{wang2025collaborative,
  author = {N. Y. Wang and Deyong Shang and Yan Gong and Xiaoxi Hu and Ziying Song and Lei Yang and Yuhan Huang and Xiaoyu Wang and Jianli Lu},
  title = {{Collaborative Perception Datasets for Autonomous Driving: A Review}},
  year = {2025},
  eprint = {2504.12696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12696},
}
Figure