Abstract
Collaborative perception allows autonomous vehicles and roadside infrastructure to share LiDAR data, significantly extending their sensing range and overcoming occlusions. However, training these systems typically requires exhaustive 3D bounding box annotations for every object in every frame, a process that is both costly and time-consuming. This paper introduces SSC3OD, a framework designed for sparsely supervised collaborative 3D object detection. By utilizing a self-supervised Pillar-based Masked Autoencoder (Pillar-MAE) and an online instance mining module, SSC3OD achieves performance levels approaching full supervision while utilizing only a fraction (~4-6%) of the total annotation data.
Introduction
Collaborative perception is a cornerstone of future intelligent transportation systems, enabling agents (vehicles and infrastructure) to "see" through obstacles by fusing multi-view LiDAR point clouds. Despite its potential, the scalability of these systems is hindered by the "supervision bottleneck." Traditional supervised learning requires dense labels, where every foreground object must be annotated. In a multi-agent environment, this requirement becomes exponentially more expensive.
When models are trained on sparse labels—for instance, when only one object per frame is annotated—they encounter the "missing label" problem. The detector learns to treat unlabeled foreground objects as background, leading to high false-negative rates and poor feature representation. SSC3OD addresses this by decoupling geometric feature learning from label dependency and implementing a robust pseudo-labeling strategy.
Methodology
The SSC3OD framework consists of two primary stages designed to mitigate the effects of incomplete supervision.
1. Pillar-based Masked Autoencoder (Pillar-MAE)
To ensure the model understands the underlying geometry of the environment without relying on human labels, the authors employ a self-supervised pre-training strategy. The LiDAR point clouds are first processed into vertical columns, or "pillars," using a PointPillars-based architecture.
The Pillar-MAE then randomly masks a high percentage (70%) of these pillars. The encoder is tasked with reconstructing the missing geometric information from the remaining 30%. This process forces the network to learn high-level semantic and structural features—such as the typical shape of a vehicle or the layout of a road—independently of any bounding box supervision.
2. Online Instance Mining Module
Following pre-training, the framework transitions to a Teacher-Student learning paradigm to handle the sparse labels.
- The Teacher: A detector trained on the initial sparse labels identifies high-confidence objects that were not included in the original manual annotations.
- The Instance Bank: These high-confidence detections are stored as pseudo-labels in an Instance Bank.
- The Student: The student model is then trained using a combination of the original sparse human labels and the mined pseudo-labels. This prevents the model from being penalized for correctly detecting real objects that were simply left unannotated.
3. Collaborative Fusion
SSC3OD is designed to be fusion-agnostic. It can be integrated with various state-of-the-art collaborative strategies, including max-pooling (F-Cooper), attention-based fusion (AttFusion), and graph-based neural networks (DiscoNet), ensuring broad applicability across different hardware and communication configurations.
Experimental Results
The researchers evaluated the framework across three major collaborative perception datasets: DAIR-V2X (real-world V2I), OPV2V (simulated V2V), and V2X-Sim (simulated V2X). To simulate a sparse environment, the authors retained only one annotated object per agent per scene, reducing the label density to approximately 4.05% to 5.90% of the original datasets.
Performance Gains
SSC3OD demonstrated significant improvements over baseline sparse models:
- V2X-Sim: Using F-Cooper fusion, the framework improved Average Precision ([email protected]) from 38.72% to 50.09%, an absolute gain of 11.37%.
- Gap Closure: Across all datasets, the framework recovered more than half of the performance gap between sparse and full supervision.
- Qualitative Observations: Visualizations indicate that while baseline sparse models frequently ignore foreground objects, SSC3OD identifies them with high confidence, producing results that closely mirror ground-truth labels.
Discussion and Conclusion
SSC3OD proves that dense manual labeling is not a prerequisite for high-performance collaborative perception. By prioritizing geometric understanding through Pillar-MAE and iteratively refining detections via instance mining, the framework effectively circumvents the "background trap" inherent in sparse supervision.
Limitations and Future Work
While SSC3OD shows promise, it currently operates under the assumption of ideal communication conditions. Future research should address the impact of communication latency and pose errors (spatial misalignment) on the feature fusion process. Additionally, the authors suggest exploring even weaker forms of supervision, such as 2D point-clicks, to further minimize the human effort required to deploy robust 3D detection systems in the real world.
Summary updated