Introduction
Cooperative perception is a cornerstone of safe and efficient autonomous driving, enabling vehicles to "see" through obstacles and beyond their immediate sensor range by sharing data with other vehicles and infrastructure. However, the primary challenge in multi-agent perception is the trade-off between communication bandwidth and detection accuracy. Existing methods typically rely on sharing dense feature maps (intermediate fusion), which, while accurate, consume significant network resources by transmitting redundant information about empty space.
CoopDETR introduces a novel, sparse, object-level communication paradigm. By shifting the focus from dense bird's-eye-view (BEV) features to a discrete set of "object queries," CoopDETR achieves state-of-the-art 3D detection performance while reducing transmission costs to just 1/782 of previous intermediate fusion methods.
The Limitations of Current Fusion Paradigms
Autonomous vehicles (AVs) currently utilize three main fusion strategies for cooperative perception:
- Early Fusion: Sharing raw sensor data. This provides the highest accuracy but is practically impossible due to extreme bandwidth requirements.
- Late Fusion: Sharing final detection results (bounding boxes). This is highly efficient but lacks the contextual nuance needed for high-precision detection in complex scenes.
- Intermediate Fusion: Sharing compressed feature maps. While the current industry standard, these maps are often redundant, lack interpretability, and are highly sensitive to localization errors (pose noise).
CoopDETR addresses these inefficiencies by asking a fundamental question: Why share the entire scene when we only care about the objects within it?
The CoopDETR Framework
CoopDETR (Cooperative Detection Transformer) utilizes a two-stage process to generate and fuse sparse object representations.
1. Single-Agent Query Generation (PointDETR)
Each agent (vehicle or infrastructure unit) processes its local LiDAR point cloud using a PointPillar encoder to generate a BEV feature map. Instead of transmitting this map, the agent employs a transformer decoder, termed PointDETR, to extract a sparse set of object queries. Each query is a high-dimensional vector representing a potential object, tied to a specific spatial reference point. This effectively filters out the "vacuum" of empty roads at the source.
2. Cross-Agent Query Fusion
When an ego vehicle receives queries from surrounding agents, it must reconcile multiple perspectives of the same environment. CoopDETR achieves this through two mechanisms:
- Spatial Query Matching (SQM): The system calculates similarity between incoming queries based on visual features and spatial coordinates. This allows the model to group queries that describe the same physical object seen from different angles.
- Object Query Aggregation (OQA): Using an attention mechanism, the system fuses these grouped queries into a single, refined representation. This allows the ego vehicle to benefit from the collective intelligence of the network without the noise of redundant data.
Experimental Results and Insights
Bandwidth and Accuracy
The researchers evaluated CoopDETR on the OPV2V and V2XSet benchmarks, both generated via the CARLA simulator. The results demonstrate a significant breakthrough in efficiency:
- Efficiency: CoopDETR reduced transmission costs to 1/782 of previous state-of-the-art models like V2X-ViT.
- Performance: On the OPV2V dataset, CoopDETR improved Average Precision ([email protected]) by 8.91% over the previous leading model, Where2comm.
Robustness to Pose Noise
In real-world scenarios, GPS and IMU errors lead to "pose noise," where vehicles are slightly misaligned in shared space. Traditional intermediate fusion methods suffer significantly from this, as misaligned feature maps lead to ghosting or missed detections. CoopDETR’s use of Deformable Attention allows the model to dynamically adjust its focus around reference points, making it inherently more resilient to localization and heading errors.
Interpretability
Unlike abstract neural feature maps, object queries are directly tied to physical instances. This provides a layer of interpretability that is crucial for safety-critical systems, allowing engineers to more easily diagnose why a system identified or overlooked a specific object based on the shared query data.
Conclusion and Future Work
CoopDETR represents a shift toward sparse, intelligent communication in multi-agent systems. By proving that object-level queries can outperform dense feature maps in both accuracy and efficiency, the framework sets a new benchmark for V2X communication.
Future research directions include moving beyond simulated CARLA environments to real-world datasets where LiDAR noise and communication latency are more volatile. Additionally, the authors suggest extending the framework to multi-modal inputs, integrating camera data into the object query generation process to create an even more robust cooperative perception stack.
Summary updated