SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles

Deyuan Qu, Qi Chen, Tianyu Bai, Hongsheng Lu, Heng Fan, Hao Zhang, Song Fu, Qing Yang

TL;DR

This paper introduces SiCP, a novel framework that enables connected and automated vehicles to perform both individual and cooperative 3D object detection simultaneously. By introducing a lightweight Dual-Perception Network (DP-Net), SiCP significantly improves cooperative perception performance while maintaining the accuracy of standalone perception, addressing a critical limitation in current approaches.

What Problem Does This Paper Address?

Connected and automated vehicles (CAVs) can enhance their perception capabilities by sharing sensor data and fusing information from multiple vehicles (cooperative perception). However, traditional cooperative perception models often suffer a significant performance drop when this shared data is unavailable, forcing them to rely solely on their own sensors (individual perception). This is problematic because CAVs may not always have reliable communication links or sufficient computational resources to run separate models for both tasks. The core issue is that existing cooperative perception models are not designed to effectively handle individual perception tasks when data sharing is not possible, leading to a performance gap. This paper aims to bridge this gap by proposing a unified framework that excels at both individual and cooperative perception.

Methods

The proposed solution is the Simultaneous Individual and Cooperative Perception (SiCP) framework. SiCP is built upon a standard 3D object detection backbone (like PointPillars) and enhances it with a novel Dual-Perception Network (DP-Net).

Here's how DP-Net works:

  • Feature Extraction: Each vehicle uses an identical feature extractor to generate local features.
  • Dual-Perception Network (DP-Net): This is the core innovation.
  • Individual Perception Path: If no features are received from other vehicles, DP-Net uses the ego vehicle's local features directly for individual perception.
  • Cooperative Perception Path: When features from neighboring vehicles are available, they are first transformed to the ego vehicle's perspective. Then, a complementary feature fusion mechanism is employed. This fusion learns a weighted map to intelligently combine the ego vehicle's features with the transformed features from other vehicles. Crucially, this fusion method is designed to preserve gradient information, which is vital for effective training and avoids the gradient vanishing issues seen in prior methods like F-Cooper.
  • Unified Detection Head: A single detection head processes the output from DP-Net, whether it's from the individual or cooperative path, generating classification and regression results.

The framework uses a "First-Come-First-Serve" (FCFS) approach for feature fusion, meaning it starts fusing as soon as features from a neighbor arrive, without waiting for all potential neighbors. This approach is designed to be lightweight and robust to alignment errors caused by asynchronous communication or localization inaccuracies between vehicles.

Datasets

The authors evaluated their framework on two widely used datasets for V2V cooperative perception research:

  • V2V4Real: A large-scale, real-world dataset collected by two vehicles simultaneously.
  • OPV2V: A simulation dataset featuring various virtual cities and environments, built using OpenCDA and CARLA.

Key Insights and Ideas

  • The critical need for dual-functionality: The paper highlights that cooperative perception models often neglect individual perception performance, which is a significant drawback in real-world scenarios where communication might be unreliable.
  • Complementary Feature Fusion: The novel fusion mechanism is key. By learning complementary weights (M and 1-M), it intelligently blends features from different sources, ensuring that important gradient information is preserved, unlike max-pooling based methods.
  • Robustness to real-world challenges: DP-Net is shown to be resilient to asynchronous communication delays and localization errors, which are common issues in V2V scenarios.
  • Lightweight and Plug-and-Play: DP-Net adds minimal parameters (0.13M) and can be easily integrated into existing standalone 3D detection models, making it highly practical for deployment.

Key Results

  • Superior Cooperative Perception: SiCP consistently outperforms state-of-the-art cooperative perception methods on both V2V4Real and OPV2V datasets. For instance, on OPV2V, SiCP achieves an AP of 71.89% at IoU=0.7, outperforming the second-best by 1.75%.
  • Competitive Individual Perception: SiCP's individual perception performance is comparable to or slightly below the best standalone models (like PointPillars), but crucially, it doesn't suffer the significant drop seen in other cooperative models when data is unavailable.
  • Robustness: SiCP maintains strong performance even with a 100ms communication delay and when localization noise is introduced, outperforming competitors by a notable margin (e.g., 8.2% AP improvement in near-range detection under asynchronous conditions).
  • Efficiency: The DP-Net module is very lightweight, adding only about 1.7% more parameters than the baseline PointPillars model, and achieves a low latency of 37.84ms.

Limitations and Future Work

  • Vehicle Trustworthiness Assumption: The framework assumes that all vehicles are using the same machine-learning model and are trustworthy. The paper does not explore scenarios with untrustworthy or heterogeneous models.
  • FCFS Limitation: While FCFS is efficient, it might not always be optimal for feature fusion compared to waiting for all available data.
  • Future Work: The authors suggest exploring more advanced fusion strategies and investigating the impact of different vehicle trust levels on cooperative perception. The paper also implies that further research into optimizing communication efficiency and handling more complex multi-agent scenarios would be beneficial.

Summary updated

Abstract

Cooperative perception for connected and automated vehicles is traditionally achieved through the fusion of feature maps from two or more vehicles. However, the absence of feature maps shared from other vehicles can lead to a significant decline in 3D object detection performance for cooperative perception models compared to standalone 3D detection models. This drawback impedes the adoption of cooperative perception as vehicle resources are often insufficient to concurrently employ two perception models. To tackle this issue, we present Simultaneous Individual and Cooperative Perception (SiCP), a generic framework that supports a wide range of the state-of-the-art standalone perception backbones and enhances them with a novel Dual-Perception Network (DP-Net) designed to facilitate both individual and cooperative perception. In addition to its lightweight nature with only 0.13M parameters, DP-Net is robust and retains crucial gradient information during feature map fusion. As demonstrated in a comprehensive evaluation on the V2V4Real and OPV2V datasets, thanks to DP-Net, SiCP surpasses state-of-the-art cooperative perception solutions while preserving the performance of standalone perception solutions. The source code can be found at https://github.com/DarrenQu/SiCP.
BibTeX
@inproceedings{qu2024sicp,
  author = {Deyuan Qu and Qi Chen and Tianyu Bai and Hongsheng Lu and Heng Fan and Hao Zhang and Song Fu and Qing Yang},
  title = {{SiCP: Simultaneous Individual and Cooperative Perception for 3D Object Detection in Connected and Automated Vehicles}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=2HIQw9fpHz},
}
Figure