TL;DR
V2X-Real is the first large-scale, real-world dataset for Vehicle-to-Everything (V2X) cooperative perception that integrates data from both multiple autonomous vehicles and smart infrastructure. With over 1.2 million 3D bounding box annotations across 33K LiDAR frames, it provides a rigorous benchmark for how connected agents can share sensor data to overcome occlusions and extend perception range in dense urban environments.
What Problem Does This Paper Address?
While autonomous vehicles (AVs) have advanced rapidly, single-vehicle perception systems are fundamentally limited by their line-of-sight. Occlusions (like a large truck blocking a car's view) and limited sensor range create dangerous "blind spots."
Cooperative perception—where vehicles (V2V) and infrastructure (V2I) share sensing data—is the proposed solution. However, research has been bottlenecked by data:
- The Simulation Gap: Most existing V2X datasets (like OPV2V or V2X-Sim) are generated in simulators, which don't perfectly capture real-world sensor noise or complex physics.
- Limited Scope: Existing real-world datasets typically focus on either V2V or V2I, but rarely both simultaneously.
- Low Complexity: Previous real-world datasets were often collected in suburban areas with low traffic density (averaging ~12 objects per scene).
V2X-Real fills these gaps by providing a multi-agent, multi-modal dataset collected in congested urban settings with high traffic density.
Methods
The researchers deployed a sophisticated hardware suite consisting of two Connected and Automated Vehicles (CAVs) and two smart infrastructure units (Infra).
- Hardware: Each vehicle carried a 128-beam LiDAR and four stereo cameras for a 360° field of view. Infrastructure units were equipped with 128/64-beam LiDARs and high-resolution network cameras.
- Synchronization & Calibration: To ensure the agents were "talking" about the same moment in time and space, the team synchronized all clocks via GPS and used dense LiDAR maps to align coordinate frames across all four agents.
- Annotation Pipeline: They used a "holistic view" approach for annotation. By aggregating LiDAR points from multiple agents, annotators could see objects that were occluded from any single viewpoint, significantly improving ground-truth accuracy.
Datasets
The authors introduce V2X-Real, which is further categorized into four sub-datasets based on the "ego" agent and collaboration mode:
- V2X-Real-VC (Vehicle-Centric): The vehicle is the primary agent, receiving data from other vehicles and infrastructure.
- V2X-Real-IC (Infrastructure-Centric): The infrastructure is the primary agent, a crucial perspective for intelligent traffic management.
- V2X-Real-V2V: Focuses purely on vehicle-to-vehicle cooperation.
- V2X-Real-I2I: Focuses on infrastructure-to-infrastructure cooperation.
Compared to predecessors like DAIR-V2X, V2X-Real features a much higher object density (average of 36 objects per scene, peaking at 115) and includes 10 categories of objects, including "vulnerable road users" like pedestrians, scooters, and cyclists.
Key Insights and Ideas
- Infrastructure as a Peer, Not Just an Aide: Unlike previous works that treat infrastructure as a secondary data source, this paper highlights Infrastructure-Centric (IC) perception. This is vital for "smart city" applications like signal control and traffic monitoring.
- The "Aha" of Combined Views: The paper demonstrates that aggregating LiDAR points from different agents (e.g., green and pink points in Fig. 3) not only makes annotation easier but reveals objects that are otherwise invisible, proving the fundamental value of V2X.
- Intermediate Fusion Wins: The benchmarks confirm that "Intermediate Fusion"—sharing compressed neural features rather than raw data (Early Fusion) or just final detections (Late Fusion)—strikes the best balance between communication bandwidth and perception accuracy.
Key Results
The authors benchmarked several State-of-the-Art (SOTA) methods, including AttFuse and V2X-ViT.
- Cooperation vs. Solo: In all scenarios, cooperative methods significantly outperformed the "No Fusion" baseline. For example, in the Vehicle-Centric (VC) task, V2X-ViT achieved a mean Average Precision (mAP) of 44.8% (at IoU 0.3), compared to only 28.2% for a single vehicle acting alone.
- Fusion Strategy Performance: Intermediate fusion methods generally outperformed Late Fusion. Interestingly, Late Fusion sometimes increased false negatives because the lack of raw context made it harder to resolve uncertainties in shared detection results.
- Urban Challenges: Performance was highest for cars but lower for pedestrians and trucks. The authors attribute this to the sparse LiDAR returns from small pedestrians and the highly variable dimensions of different truck types.
Limitations and Future Work
- Sensor Diversity: The current dataset is limited to LiDAR and cameras. The authors acknowledge that adding Radar is a critical next step for developing robust, low-cost V2X solutions that can handle adverse weather.
- Infrastructure Topology: Infrastructure cameras have fixed positions relative to the road. The authors suggest that future work should develop specialized algorithms that leverage this fixed topology as a "prior" to improve detection efficiency.
- Scalability: The team plans to extend the dataset to include lower-cost sensors (like low-beam LiDAR) to study the feasibility of mass-market V2X deployment.
The V2X-Real dataset and codebase are open-sourced at https://mobility-lab.seas.ucla.edu/v2x-real.
Summary updated