Rethinking Collaborative Perception from the Spatial-Temporal Importance of Semantic Information

Yuntao Liu, Qian Huang, Rongpeng Li, Xianfu Chen, Zhifeng Zhao, Shuyuan Zhao, Yongdong Zhu, Honggang Zhang

Abstract

Collaborative perception is a promising approach to enhance the safety of autonomous vehicles (AVs) by enabling them to share sensor data via Vehicle-to-Everything (V2X) communication. While existing methods focus on spatial feature fusion, they often ignore the critical impact of temporal latency and sensor asynchrony. This paper introduces Select2Col, a novel framework that evaluates the Importance of Semantic Information (IoSI) based on both spatial and temporal factors. By intelligently selecting collaborators and utilizing a historical prior hybrid attention mechanism, Select2Col achieves state-of-the-art performance in 3D object detection while maintaining robustness against communication delays.

Introduction: The Challenge of Perceptual Blindness

Autonomous vehicles rely heavily on local sensors like LiDAR and cameras. However, these sensors are limited by line-of-sight occlusions—such as a large truck blocking the view of a pedestrian—and finite detection ranges. Collaborative perception addresses these "blind spots" by allowing vehicles and roadside infrastructure to share extracted features, known as semantic information.

Collaborative perception scenario where an ego vehicle receives semantic information from collaborators to detect potential collisions hidden by occlusions.

Despite the potential of V2X, real-world communication is rarely perfect. Transmission delays and varying sensor sampling rates mean that the information received by an "ego" vehicle may be outdated (stale). Fusing this stale data without correction can lead to "ghost" detections or misaligned object positions, ultimately compromising safety. Select2Col is designed to address this spatial-temporal mismatch.

Methodology: The Select2Col Framework

The Select2Col architecture is a modular pipeline designed for efficiency and accuracy. It processes raw LiDAR point clouds through four main stages: Encoding, Selection, Fusion, and Decoding.

The Select2Col architecture overview, illustrating the flow from raw point clouds through the Encoder, Selector, Fuser (HPHA), and Decoder modules.

1. Encoder

The system uses a PointPillars backbone to convert raw 3D point clouds into a 2D representation. This approach is computationally efficient, making it suitable for real-time processing on vehicle hardware.

2. IoSI-based Collaborator Selection

Rather than fusing data from every available agent, Select2Col employs a Selector module. This module uses a lightweight Graph Neural Network (GNN) to calculate the Importance of Semantic Information (IoSI). The GNN considers the spatial coverage of a collaborator's data and its associated communication latency. By downweighting or discarding irrelevant or excessively delayed information, the system reduces noise and optimizes bandwidth usage.

3. HPHA (Historical Prior Hybrid Attention) Fusion

The Fuser module merges the selected features using a two-step attention mechanism:

  • Multi-scale Spatial Attention: This module captures correlations between the ego vehicle and its collaborators across different spatial resolutions, ensuring that both fine-grained and global features are preserved.
  • Short-term Temporal Attention: This component uses the ego vehicle’s own historical frames as a reference. By comparing incoming collaborative data against its own recent memory, the system can filter out temporal noise and better align stale data with the current environment.

4. Decoder

Finally, the fused information is passed to a decoder that performs classification and bounding box regression to produce the final 3D object detection results.

Evaluation and Results

Select2Col was rigorously tested on three benchmark datasets: OPV2V (simulation), V2XSet (V2I scenarios), and V2V4Real (real-world data).

  • Superior Accuracy: On the OPV2V dataset, Select2Col achieved an Average Precision (AP) of 77.65% (at IoU=0.7), outperforming established models like V2VNet (52.96%) and Where2comm.
  • Real-World Performance: In the V2V4Real benchmark, which features noisy real-world environments, the model maintained a significant lead with 24.89% AP.
  • Resilience to Latency: One of the most critical findings was the model's stability under high latency. Even with a simulated delay of 400ms, Select2Col’s performance remained robust, whereas competing models experienced sharp declines in accuracy.
  • Efficiency: With only 8.28 million parameters and an inference time under 100ms, the framework is well-suited for the 10Hz cycle typical of automotive LiDAR systems.

Discussion and Future Work

Select2Col demonstrates that "when" information is recorded is just as important as "where" it comes from. By treating latency as a core variable rather than an external nuisance, the framework achieves a more reliable perception of the environment.

However, challenges remain. The current weighting strategy is sensitive to specific detection thresholds, and the sharing of semantic features raises potential privacy concerns. Future research will focus on refining these selection thresholds and implementing privacy-preserving techniques to prevent the reconstruction of sensitive data from shared feature maps.

Summary updated

BibTeX
@article{liu2024rethinking,
  author = {Yuntao Liu and Qian Huang and Rongpeng Li and Xianfu Chen and Zhifeng Zhao and Shuyuan Zhao and Yongdong Zhu and Honggang Zhang},
  title = {{Rethinking Collaborative Perception from the Spatial-Temporal Importance of Semantic Information}},
  journal = {IEEE Trans. Vehicular Technology},
  year = {2024},
  doi = {10.1109/tvt.2024.3390414},
}
Figure