You Share Beliefs, I Adapt: Progressive Heterogeneous Collaborative Perception

Hao Si, Ehsan Javanmardi, Manabu TsukadaICCV 2025

TL;DR

This paper introduces Progressive Heterogeneous Collaborative Perception (PHCP), a novel framework that enables autonomous vehicles to adapt to each other's perception models during inference, eliminating the need for costly joint training. By leveraging few-shot unsupervised domain adaptation and self-training with pseudo-labels, PHCP significantly improves collaborative perception performance in heterogeneous environments, achieving results comparable to state-of-the-art methods that require extensive training.

What Problem Does This Paper Address?

Autonomous driving systems are increasingly leveraging Vehicle-to-Everything (V2X) communication for collaborative perception, allowing vehicles to share information and overcome individual limitations like occlusions. However, a major real-world challenge is heterogeneity: vehicles from different manufacturers often have different sensor configurations and perception models, leading to a "domain gap" in their intermediate feature representations. Existing solutions typically require joint training or pre-defined adapters for each potential collaborator, which is impractical as new vehicle models emerge. This paper tackles the critical question: Can we achieve effective heterogeneous collaborative perception during inference, without prior knowledge or joint training?

Methods

The core of PHCP is a two-stage process designed for inference-time adaptation:

  • Stage I: Ego Fine-tuning by Self-Training: In the initial few frames (k frames) after a collaboration is established, the ego vehicle receives both features and detection results from other agents. The ego vehicle then uses these results as "pseudo-labels" to fine-tune a lightweight feature adapter. This adapter is trained using a few-shot unsupervised domain adaptation approach, specifically leveraging a two-stage fine-tuning strategy. The adapter is designed to be simple (using CBAM for channel and spatial attention) to prevent overfitting and minimize computational overhead.
  • Stage II: Ego Inference: After the initial adaptation, the ego vehicle only needs to receive intermediate features from collaborating agents. It then uses the fine-tuned adapter to align these features to its own domain before performing fusion and final detection. This approach effectively bridges the domain gap dynamically during the collaboration.

Datasets

  • OPV2V: Used for training baseline models and evaluating the proposed PHCP framework.
  • OPV2V-H: A supplementary dataset used for evaluation.

The paper utilizes a split of the OPV2V dataset into support sets (S) and query sets (Q) for the few-shot learning aspect of Stage I.

Key Insights and Ideas

  • Inference-time Adaptation is Key: The most significant insight is that adapting perception models during inference is a viable and practical solution for heterogeneous collaboration, bypassing the limitations of traditional joint training.
  • Pseudo-labels as Supervision: Leveraging the predictions of other agents as pseudo-labels for self-training the adapter is an effective way to generate supervision signals in an unsupervised inference setting.
  • Lightweight Adapters are Sufficient: The paper demonstrates that a simple, lightweight adapter (like CBAM) can effectively bridge the domain gap without requiring extensive architectural changes or computational resources.
  • Few-Shot Learning for Efficiency: The few-shot approach significantly reduces the data and time required for adaptation, making it suitable for real-time applications.
  • Domain Gap is Primarily Feature Misalignment: Visualizations show that the domain gap is not a complete disconnect but rather a misalignment in feature distributions, particularly in channel dimensions and critical regions, which attention mechanisms can address.

Key Results

  • Significant Performance Improvement: PHCP consistently outperforms the direct fusion baseline (without joint training) by a substantial margin, achieving improvements of 33.2%, 32.9%, and 32.9% in [email protected], [email protected], and [email protected], respectively.
  • Comparable to SOTA: The proposed method achieves performance comparable to state-of-the-art methods that are trained on the entire dataset with labels, while using only a small amount of unlabeled data for adaptation.
  • Robustness Across Scenarios: Experiments show that PHCP maintains strong performance across diverse heterogeneous scenarios, even those where the baseline struggles.
  • Efficient Inference: The inference time is very low (0.07 seconds), meeting real-time requirements. Training time for the adapter is also minimal (1.5-2.4 seconds depending on shots).
  • Qualitative Improvements: Visualizations show PHCP successfully detecting objects outside the ego vehicle's field of view, which the baseline misses, and improved feature alignment.

Limitations and Future Work

  • Number of Shots Trade-off: While more shots (more data for adaptation) improve performance, they increase the time to initiate collaboration. The paper explores this trade-off, showing significant gains even with few shots.
  • Pseudo-label Quality: The quality of pseudo-labels is crucial. The paper investigates filtering strategies, finding that retaining high-quality labels is important, though even lower-quality labels can be managed by the adapter.
  • Generalization Across Scenarios: While the model shows good generalization, performance slightly declines in some complex scenarios. Future work could explore more robust cross-scenario generalization techniques.
  • Future Work: The authors suggest further research into more advanced adaptation strategies and exploring the impact of different adapter architectures. The paper also implicitly points towards the need for standardized benchmarks for evaluating heterogeneous collaborative perception.

Summary updated

BibTeX
@inproceedings{si2025you,
  author = {Hao Si and Ehsan Javanmardi and Manabu Tsukada},
  title = {{You Share Beliefs, I Adapt: Progressive Heterogeneous Collaborative Perception}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure