V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception

Hao Xiang, Zhaoliang Zheng, Xin Xia, Runsheng Xu, Letian Gao, Zewei Zhou, Xu Han, Xinkai Ji, Mingxi Li, Zonglin Meng, Li Jin, Mingyue Lei, Zhaoyang Ma, Zihang He, Haoxuan Ma, Yunshuang Yuan, Yingqian Zhao, Jiaqi Ma

TL;DR

V2X-Real is the first large-scale, real-world dataset for Vehicle-to-Everything (V2X) cooperative perception that integrates data from both multiple autonomous vehicles and smart infrastructure. With over 1.2 million 3D bounding box annotations across 33K LiDAR frames, it provides a rigorous benchmark for how connected agents can share sensor data to overcome occlusions and extend perception range in dense urban environments.

What Problem Does This Paper Address?

While autonomous vehicles (AVs) have advanced rapidly, single-vehicle perception systems are fundamentally limited by their line-of-sight. Occlusions (like a large truck blocking a car's view) and limited sensor range create dangerous "blind spots."

Cooperative perception—where vehicles (V2V) and infrastructure (V2I) share sensing data—is the proposed solution. However, research has been bottlenecked by data:

  1. The Simulation Gap: Most existing V2X datasets (like OPV2V or V2X-Sim) are generated in simulators, which don't perfectly capture real-world sensor noise or complex physics.
  2. Limited Scope: Existing real-world datasets typically focus on either V2V or V2I, but rarely both simultaneously.
  3. Low Complexity: Previous real-world datasets were often collected in suburban areas with low traffic density (averaging ~12 objects per scene).

V2X-Real fills these gaps by providing a multi-agent, multi-modal dataset collected in congested urban settings with high traffic density.

Methods

The researchers deployed a sophisticated hardware suite consisting of two Connected and Automated Vehicles (CAVs) and two smart infrastructure units (Infra).

  • Hardware: Each vehicle carried a 128-beam LiDAR and four stereo cameras for a 360° field of view. Infrastructure units were equipped with 128/64-beam LiDARs and high-resolution network cameras.
  • Synchronization & Calibration: To ensure the agents were "talking" about the same moment in time and space, the team synchronized all clocks via GPS and used dense LiDAR maps to align coordinate frames across all four agents.
  • Annotation Pipeline: They used a "holistic view" approach for annotation. By aggregating LiDAR points from multiple agents, annotators could see objects that were occluded from any single viewpoint, significantly improving ground-truth accuracy.
Fig. 1: Demonstration of smart intersection and overall data acquisition systems. Smart infrastructure (Infra) units and connected and automated vehicles (CAVs) are depicted in their combined LiDAR point clouds.

Datasets

The authors introduce V2X-Real, which is further categorized into four sub-datasets based on the "ego" agent and collaboration mode:

  • V2X-Real-VC (Vehicle-Centric): The vehicle is the primary agent, receiving data from other vehicles and infrastructure.
  • V2X-Real-IC (Infrastructure-Centric): The infrastructure is the primary agent, a crucial perspective for intelligent traffic management.
  • V2X-Real-V2V: Focuses purely on vehicle-to-vehicle cooperation.
  • V2X-Real-I2I: Focuses on infrastructure-to-infrastructure cooperation.

Compared to predecessors like DAIR-V2X, V2X-Real features a much higher object density (average of 36 objects per scene, peaking at 115) and includes 10 categories of objects, including "vulnerable road users" like pedestrians, scooters, and cyclists.

Key Insights and Ideas

  • Infrastructure as a Peer, Not Just an Aide: Unlike previous works that treat infrastructure as a secondary data source, this paper highlights Infrastructure-Centric (IC) perception. This is vital for "smart city" applications like signal control and traffic monitoring.
  • The "Aha" of Combined Views: The paper demonstrates that aggregating LiDAR points from different agents (e.g., green and pink points in Fig. 3) not only makes annotation easier but reveals objects that are otherwise invisible, proving the fundamental value of V2X.
  • Intermediate Fusion Wins: The benchmarks confirm that "Intermediate Fusion"—sharing compressed neural features rather than raw data (Early Fusion) or just final detections (Late Fusion)—strikes the best balance between communication bandwidth and perception accuracy.
Fig. 3: Demonstration of combined LiDAR annotations for infrastructure. Li- DAR points from different agents (green and pink) are aggregated for the same object, reducing half of the infrastructure annotation workload and enhancing annotators’ ability to identify objects accurately and efficiently.

Key Results

The authors benchmarked several State-of-the-Art (SOTA) methods, including AttFuse and V2X-ViT.

  • Cooperation vs. Solo: In all scenarios, cooperative methods significantly outperformed the "No Fusion" baseline. For example, in the Vehicle-Centric (VC) task, V2X-ViT achieved a mean Average Precision (mAP) of 44.8% (at IoU 0.3), compared to only 28.2% for a single vehicle acting alone.
  • Fusion Strategy Performance: Intermediate fusion methods generally outperformed Late Fusion. Interestingly, Late Fusion sometimes increased false negatives because the lack of raw context made it harder to resolve uncertainties in shared detection results.
  • Urban Challenges: Performance was highest for cars but lower for pedestrians and trucks. The authors attribute this to the sparse LiDAR returns from small pedestrians and the highly variable dimensions of different truck types.

Limitations and Future Work

  • Sensor Diversity: The current dataset is limited to LiDAR and cameras. The authors acknowledge that adding Radar is a critical next step for developing robust, low-cost V2X solutions that can handle adverse weather.
  • Infrastructure Topology: Infrastructure cameras have fixed positions relative to the road. The authors suggest that future work should develop specialized algorithms that leverage this fixed topology as a "prior" to improve detection efficiency.
  • Scalability: The team plans to extend the dataset to include lower-cost sensors (like low-beam LiDAR) to study the feasibility of mass-market V2X deployment.

The V2X-Real dataset and codebase are open-sourced at https://mobility-lab.seas.ucla.edu/v2x-real.

Summary updated

Abstract

Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting the perception capability. However, there are no real-world datasets to facilitate the real V2X cooperative perception research -- existing datasets either only support Vehicle-to-Infrastructure cooperation or Vehicle-to-Vehicle cooperation. In this paper, we present V2X-Real, a large-scale dataset that includes a mixture of multiple vehicles and smart infrastructure to facilitate the V2X cooperative perception development with multi-modality sensing data. Our V2X-Real is collected using two connected automated vehicles and two smart infrastructure, which are all equipped with multi-modal sensors including LiDAR sensors and multi-view cameras. The whole dataset contains 33K LiDAR frames and 171K camera data with over 1.2M annotated bounding boxes of 10 categories in very challenging urban scenarios. According to the collaboration mode and ego perspective, we derive four types of datasets for Vehicle-Centric, Infrastructure-Centric, Vehicle-to-Vehicle, and Infrastructure-to-Infrastructure cooperative perception. Comprehensive multi-class multi-agent benchmarks of SOTA cooperative perception methods are provided. The V2X-Real dataset and codebase are available at https://mobility-lab.seas.ucla.edu/v2x-real.
BibTeX
@inproceedings{xiang2024v,
  author = {Hao Xiang and Zhaoliang Zheng and Xin Xia and Runsheng Xu and Letian Gao and Zewei Zhou and Xu Han and Xinkai Ji and Mingxi Li and Zonglin Meng and Li Jin and Mingyue Lei and Zhaoyang Ma and Zihang He and Haoxuan Ma and Yunshuang Yuan and Yingqian Zhao and Jiaqi Ma},
  title = {{V2X-Real: a Large-Scale Dataset for Vehicle-to-Everything Cooperative Perception}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72943-0\_26},
}
Figure