Introduction
In the domain of autonomous driving, robust perception requires a unified understanding of the environment across multiple sensors. Traditionally, camera images (RGB) and LiDAR point clouds are processed as independent data streams. While cameras provide rich texture and long-range semantic information, LiDAR offers precise 3D geometric data. This paper introduces a novel framework for instance segmentation that bridges the gap between these modalities by learning a shared, consistent embedding space.
Methodology
The core of the approach is the learning of dense embeddings—mathematical representations assigned to every pixel and LiDAR point. The objective is to ensure that all points belonging to the same object instance have similar embeddings, regardless of the sensor modality or the time of capture.
Cross-Modal Contrastive Loss
The system is trained using a cross-modal contrastive loss, a variation of the NT-XENT (Normalized Temperature-scaled Cross Entropy) loss. This loss function supervises the model to:
- Cluster embeddings of the same instance together.
- Separate embeddings of different instances. This consistency is enforced spatially within frames, across modalities (RGB vs. LiDAR), and temporally across video sequences.
Temporal Pseudo-Ground Truth
To facilitate training on large-scale video data without exhaustive manual labeling, the authors employ a temporal "warping" technique. Using a pre-trained optical flow model, instance labels from a single annotated frame are propagated to neighboring unlabeled frames. This creates "pseudo-labels" that provide a training signal for temporal stability.
Inference via Clustering
At runtime, the network predicts dense embeddings for the input data. These embeddings are then grouped into distinct object masks using Mean Shift Clustering. This "bottom-up" approach is computationally efficient and avoids the need for region proposals used in "top-down" methods like Mask R-CNN.
Experimental Evaluation
The framework was evaluated on two major benchmarks:
- KITTI-360: A comprehensive dataset for mobile 2D/3D perception with synchronized RGB and LiDAR data.
- Cityscapes: A standard urban scene dataset used to test temporal consistency in single-modal (RGB) settings.
Key Results and Insights
The research demonstrates that cross-modal consistency serves as an effective self-supervisory signal, improving the performance of individual sensors by forcing them to "agree" on object identities.
- Accuracy Improvements: On the KITTI-360 benchmark, the addition of temporal RGB consistency improved mean Average Precision (mAP) by 1.9 points, while cross-modal consistency (LiDAR + RGB) added another 1.7 mAP.
- Temporal Reliability: The method significantly enhanced temporal stability, reducing embedding variance over time and improving temporal accuracy from 75.1% to 83.4%.
- Robustness to Rare Classes: The model showed marked improvements in segmenting safety-critical but less frequent classes, such as pedestrians and cyclists.
Conclusion
By enforcing consistency across sensors and time, this approach creates a more reliable and accurate foundation for instance segmentation in autonomous systems. The learned embeddings are not only more accurate but also more stable, providing a "stickier" identity for objects as they move through a scene. Future work will explore replacing the simple clustering post-processing with more advanced architectures like Transformers to further push the boundaries of segmentation performance.
Summary updated