Abstract
V2X-ViT is the first unified Vision Transformer framework designed for Vehicle-to-Everything (V2X) cooperative perception. By fusing data from both vehicles and roadside infrastructure, it overcomes the "blind spots" of single-vehicle systems and maintains high performance even in the face of real-world challenges like GPS errors and network delays.
Introduction
While autonomous vehicles (AVs) have become remarkably good at sensing their immediate surroundings, they still suffer from "tunnel vision." A single vehicle's sensors cannot see through large obstacles or around blind corners, and LiDAR points become sparse at long distances.
Cooperative perception—where vehicles share data—is the proposed solution. However, most existing research focuses on Vehicle-to-Vehicle (V2V) communication, ignoring the massive potential of roadside infrastructure. Furthermore, real-world V2X is complex due to sensor heterogeneity, localization (GPS) errors, and transmission delays. V2X-ViT addresses these "messy" variables using a robust Transformer-based architecture.
System Architecture
The V2X-ViT pipeline starts by extracting features using PointPillars, compressing them for transmission, and then passing them through a specialized fusion engine. Unlike previous methods that struggle with spatial misalignment, V2X-ViT uses attention mechanisms to adaptively integrate multi-agent data.
Core Innovations in Attention
The heart of the framework lies in two novel attention modules designed to handle heterogeneity and noise:
Heterogeneous Multi-agent Self-Attention (HMSA)
HMSA recognizes that the relationship between "Vehicle-to-Vehicle" is different from "Infrastructure-to-Vehicle." It uses specific weights for different types of agents and connections to better integrate the superior, high-angle views provided by infrastructure sensors.
Multi-scale Window Attention (MSwin)
To combat localization (pose) errors, the model uses a pyramid of window sizes. Large windows capture long-range visual cues to compensate for large GPS jitters, while smaller windows preserve fine-grained local details. A "Split-Attention" module then adaptively decides which scale to trust.
Delay-aware Positional Encoding (DPE)
To handle time lags, the authors use a time-conditioned sinusoid encoding that helps the model "re-align" features that arrived late due to network latency.
The V2XSet Dataset
Finding existing datasets lacking in realistic V2X noise, the researchers introduced V2XSet, a large-scale simulation dataset created using CARLA and OpenCDA. It includes 11,447 frames across 5 roadway types and explicitly simulates pose errors and transmission delays (up to 400ms), making it the first benchmark to include both vehicles and infrastructure as active participants.
Key Results and Insights
V2X-ViT sets a new state-of-the-art for 3D object detection in cooperative settings:
- Performance: In "Perfect" settings, it achieves 0.712 AP (Average Precision) at IoU=0.7, outperforming previous leaders like DiscoNet.
- Robustness: In "Noisy" settings (with pose errors and delay), V2X-ViT maintains an AP of 0.614, while other methods drop significantly.
- Efficiency: The model is real-time capable; a smaller version (V2X-ViT_S) runs in just 28ms.
The study highlights that infrastructure is often the "MVP" of the system, providing a "God's eye view" with significantly less occlusion than vehicle-mounted sensors.
Conclusion and Future Work
V2X-ViT demonstrates that multi-scale attention is a powerful tool for robustness, allowing the system to "look around" areas of spatial uncertainty. While currently validated in simulation, future directions include moving toward multi-sensor fusion (LiDAR and cameras) and exploring joint perception and prediction to better plan for future vehicle movements in dense traffic.
Summary updated