V2X-Real: a Largs-Scale Dataset for Vehicle-to-Everything Cooperative Perception

Hao Xiang, Zhaoliang Zheng, Xin Xia, Runsheng Xu, Letian Gao, Zewei Zhou, Xu Han, Xinkai Ji, Mingxi Li, Zonglin Meng, Li Jin, Mingyue Lei, Zhaoyang Ma, Zihang He, Haoxuan Ma, Yunshuang Yuan, Yingqian Zhao, Jiaqi Ma

Abstract

Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting perception capability. However, real-world datasets for comprehensive V2X cooperative perception—integrating both Vehicle-to-Infrastructure (V2I) and Vehicle-to-Vehicle (V2V) cooperation—have been missing. We propose V2X-Real, a dataset featuring a mixture of multiple vehicles and smart infrastructure units equipped with multi-modal sensors. V2X-Real contains 33K LiDAR frames and 171K camera images with over 1.2M annotated bounding boxes across 10 categories in challenging urban scenarios. We provide benchmarks for four cooperation modes: Vehicle-Centric, Infrastructure-Centric, V2V, and I2I, demonstrating significant performance gains through cooperative sensing.

Overcoming the Line-of-Sight Barrier

Single-vehicle perception systems are fundamentally limited by their line-of-sight. In dense urban environments, occlusions from large vehicles or buildings and the limited range of onboard sensors create dangerous "blind spots." Cooperative perception—where agents share sensor data—offers a solution by allowing vehicles to "see" through obstacles.

While simulation has provided initial insights, the "sim-to-real" gap remains a significant hurdle. Previous real-world datasets have typically focused on either V2V or V2I in isolation, often in low-density suburban areas. V2X-Real bridges this gap by providing a large-scale, multi-agent, multi-modal dataset collected in complex urban intersections.

The V2X-Real Hardware Suite

To capture "true" V2X data, the research team deployed a sophisticated hardware configuration consisting of two Connected and Automated Vehicles (CAVs) and two Smart Infrastructure units.

Overview of the V2X-Real hardware setup and multi-agent sensor fusion.

The CAVs are equipped with 128-beam LiDARs and four stereo cameras providing a 360° field of view. The infrastructure units utilize 128/64-beam LiDARs and high-resolution network cameras. All agents are synchronized via GPS-locked LiDAR phases and calibrated into a unified coordinate system, ensuring spatial and temporal consistency across the shared data.

Dataset Composition and Urban Complexity

V2X-Real stands out for its scale and density. It includes 1.2 million 3D bounding boxes, covering not just vehicles but also vulnerable road users (VRUs) like pedestrians, cyclists, and scooter riders.

Multi-view camera data from the V2X-Real dataset in a dense urban environment.

The dataset features an average of 36 objects per scene, significantly higher than previous real-world datasets. This high density is critical for testing perception algorithms in the chaotic environments where cooperative sensing is most needed. From this core data, four specialized sub-datasets were derived to evaluate different ego perspectives: Vehicle-Centric (VC), Infrastructure-Centric (IC), Vehicle-to-Vehicle (V2V), and Infrastructure-to-Infrastructure (I2I).

Benchmarking Cooperative Fusion

The authors evaluated four primary fusion strategies: No Fusion (single-agent baseline), Late Fusion (sharing bounding boxes), Early Fusion (sharing raw point clouds), and Intermediate Fusion (sharing compressed neural features).

The results demonstrate a massive "cooperation gain." In the Vehicle-Centric task, the state-of-the-art V2X-ViT model achieved a mean Average Precision (mAP) of 44.8%, a substantial improvement over the 28.2% achieved by the single-vehicle baseline. Intermediate fusion proved to be the most effective, balancing bandwidth efficiency with the preservation of spatial context.

Interestingly, the Infrastructure-to-Infrastructure (I2I) mode showed the highest overall accuracy (62.7% mAP), highlighting the advantage of high-vantage, stable sensor placements. However, detecting pedestrians remains a challenge, with mAPs significantly lower than those for vehicles, pointing to a key area for future research.

Conclusion

V2X-Real provides the first comprehensive real-world foundation for V2X cooperative perception research. By including both vehicles and infrastructure in high-density urban settings, it offers a rigorous benchmark for the next generation of autonomous systems. The dataset and benchmark codes are publicly available to the research community.

Summary updated

Abstract

Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting the perception capability. However, there are no real-world datasets to facilitate the real V2X cooperative perception research -- existing datasets either only support Vehicle-to-Infrastructure cooperation or Vehicle-to-Vehicle cooperation. In this paper, we propose a dataset that has a mixture of multiple vehicles and smart infrastructure simultaneously to facilitate the V2X cooperative perception development with multi-modality sensing data. Our V2X-Real is collected using two connected automated vehicles and two smart infrastructures, which are all equipped with multi-modal sensors including LiDAR sensors and multi-view cameras. The whole dataset contains 33K LiDAR frames and 171K camera data with over 1.2M annotated bounding boxes of 10 categories in very challenging urban scenarios. According to the collaboration mode and ego perspective, we derive four types of datasets for Vehicle-Centric, Infrastructure-Centric, Vehicle-to-Vehicle, and Infrastructure-to-Infrastructure cooperative perception. Comprehensive multi-class multi-agent benchmarks of SOTA cooperative perception methods are provided. The V2X-Real dataset and benchmark codes will be released.
BibTeX
@inproceedings{xiang2024v,
  author = {Hao Xiang and Zhaoliang Zheng and Xin Xia and Runsheng Xu and Letian Gao and Zewei Zhou and Xu Han and Xinkai Ji and Mingxi Li and Zonglin Meng and Li Jin and Mingyue Lei and Zhaoyang Ma and Zihang He and Haoxuan Ma and Yunshuang Yuan and Yingqian Zhao and Jiaqi Ma},
  title = {{V2X-Real: a Largs-Scale Dataset for Vehicle-to-Everything Cooperative Perception}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72943-0\_26},
}
Figure