Select2Col: Leveraging Spatial-Temporal Importance of Semantic Information for Efficient Collaborative Perception

Yuntao Liu, Qian Huang, Rongpeng Li, Xianfu Chen, Zhifeng Zhao, Shuyuan Zhao, Yongdong Zhu, Honggang Zhang

TL;DR

This paper introduces Select2Col, a novel framework for collaborative perception in autonomous vehicles. It addresses the limitations of existing methods by intelligently selecting which neighboring vehicles to collaborate with and how to fuse their sensory data, considering both spatial and temporal importance. This leads to significantly improved perception accuracy and efficiency.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) rely heavily on perception systems to understand their surroundings. However, individual AVs face challenges with occluded or distant objects, leading to potential safety risks. Collaborative perception aims to overcome these limitations by allowing AVs to share and fuse sensory information. While current methods often focus on fusing spatial features, they neglect the crucial temporal aspect of this information. This can lead to using outdated or irrelevant data, potentially degrading performance. Furthermore, not all collaborators contribute equally; some might even introduce noise due to latency or misaligned viewpoints. Therefore, there's a need for a system that can intelligently select the most beneficial collaborators and fuse their information effectively, considering both spatial and temporal dimensions.

Methods

Select2Col is built upon four core components:

  • Encoder: This component extracts semantic information from the raw sensor data (specifically LiDAR point clouds in this work). The paper uses the Pointpillars backbone for its efficiency.
  • Selector: This is a key innovation. Instead of blindly fusing information from all neighbors, Select2Col uses a lightweight Graph Neural Network (GNN) to estimate the "Importance of Semantic Information" (IoSI) from both spatial and temporal perspectives. This allows it to identify and select only the most contributive collaborators, pruning those that might introduce noise or outdated information.
  • Fuser (HPHA): The paper proposes a novel fusion algorithm called HPHA (Historical Prior Hybrid Attention). HPHA leverages a multi-scale attention module to capture spatial correlations across different resolutions and a short-term attention module to incorporate historical ego-vehicle data. This hybrid approach ensures that the fused information is both spatially and temporally relevant.
  • Decoder: This component takes the fused semantic information and generates the final perception results (e.g., 3D object detection).

The entire framework is trained end-to-end.

Datasets

The experiments were conducted on three open collaborative perception datasets:

  • OPV2V: A large-scale dataset specifically for vehicle-to-vehicle collaborative perception, generated using CARLA and SUMO.
  • V2XSet: Focuses on vehicle-to-vehicle and vehicle-to-infrastructure collaboration, also generated using CARLA and OpenCDA.
  • V2V4Real: A real-world dataset collected over three days in Columbus, Ohio, covering highway and city driving.

Key Insights and Ideas

  • Temporal Importance Matters: The paper strongly emphasizes that neglecting the temporal dimension of semantic information is a significant drawback in current collaborative perception systems. Outdated information can be detrimental.
  • Intelligent Collaborator Selection: Not all collaborators are helpful. Select2Col's ability to identify and select "contributive" collaborators based on their spatial-temporal IoSI is crucial for improving performance and reducing noise.
  • Hybrid Attention for Fusion: The HPHA fusion algorithm effectively combines spatial and temporal information by using multi-scale attention for spatial context and short-term attention with historical data for temporal relevance.
  • IoSI as a Guiding Principle: The concept of "Importance of Semantic Information" (IoSI) is central to both collaborator selection and feature fusion, ensuring that the system prioritizes the most valuable information.

Key Results

  • Superior Perception Accuracy: Select2Col consistently outperforms state-of-the-art methods like V2VNet, V2X-Vit, and Where2comm across all tested datasets and IoU thresholds. For example, on the OPV2V dataset at IoU 0.7, Select2Col achieved a significant gain of 24.69% AP compared to V2VNet.
  • Robustness to Noise: The framework demonstrates remarkable robustness to localization noise, maintaining higher AP scores compared to other methods.
  • Effective Long-Distance Perception: Select2Col excels at perceiving distant objects, showing significant improvements in AP at longer ranges.
  • Efficiency: Despite incorporating more sophisticated temporal and spatial reasoning, Select2Col achieves competitive inference times, often outperforming other methods while offering superior accuracy. Its parameter count is also comparable to or lower than many SOTA approaches.

Limitations and Future Work

  • IoU Threshold in Weighting: The current approach for calculating enhanced weights overlooks the IoU threshold, which might lead to suboptimal collaborator selection when detection difficulty varies. The authors suggest a more accurate determination method is needed.
  • Privacy Concerns: The sharing of perception information inherently carries privacy risks. The paper suggests investigating collaborative privacy protection techniques to encourage participation.
  • Further IoSI Refinement: While Select2Col significantly improves IoSI estimation, there's always room for further refinement in how IoSI is calculated and utilized.

Summary updated

BibTeX
@article{liu2024select,
  author = {Yuntao Liu and Qian Huang and Rongpeng Li and Xianfu Chen and Zhifeng Zhao and Shuyuan Zhao and Yongdong Zhu and Honggang Zhang},
  title = {{Select2Col: Leveraging Spatial-Temporal Importance of Semantic Information for Efficient Collaborative Perception}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2024},
  doi = {10.1109/tvt.2024.3390414},
}
Figure