Instance Segmentation with Cross-Modal Consistency

Alex Zihao Zhu, Vincent Casser, Reza Mahjourian, Henrik Kretzschmar, Sören Pirk

Introduction

In the domain of autonomous driving, robust perception requires a unified understanding of the environment across multiple sensors. Traditionally, camera images (RGB) and LiDAR point clouds are processed as independent data streams. While cameras provide rich texture and long-range semantic information, LiDAR offers precise 3D geometric data. This paper introduces a novel framework for instance segmentation that bridges the gap between these modalities by learning a shared, consistent embedding space.

Methodology

The core of the approach is the learning of dense embeddings—mathematical representations assigned to every pixel and LiDAR point. The objective is to ensure that all points belonging to the same object instance have similar embeddings, regardless of the sensor modality or the time of capture.

Cross-Modal Contrastive Loss

The system is trained using a cross-modal contrastive loss, a variation of the NT-XENT (Normalized Temperature-scaled Cross Entropy) loss. This loss function supervises the model to:

  1. Cluster embeddings of the same instance together.
  2. Separate embeddings of different instances. This consistency is enforced spatially within frames, across modalities (RGB vs. LiDAR), and temporally across video sequences.

Temporal Pseudo-Ground Truth

To facilitate training on large-scale video data without exhaustive manual labeling, the authors employ a temporal "warping" technique. Using a pre-trained optical flow model, instance labels from a single annotated frame are propagated to neighboring unlabeled frames. This creates "pseudo-labels" that provide a training signal for temporal stability.

Inference via Clustering

At runtime, the network predicts dense embeddings for the input data. These embeddings are then grouped into distinct object masks using Mean Shift Clustering. This "bottom-up" approach is computationally efficient and avoids the need for region proposals used in "top-down" methods like Mask R-CNN.

The training pipeline (a) utilizes a global contrastive loss across LiDAR and RGB modalities, while the inference pipeline (b) uses clustering to generate instance masks.

Experimental Evaluation

The framework was evaluated on two major benchmarks:

  • KITTI-360: A comprehensive dataset for mobile 2D/3D perception with synchronized RGB and LiDAR data.
  • Cityscapes: A standard urban scene dataset used to test temporal consistency in single-modal (RGB) settings.

Key Results and Insights

The research demonstrates that cross-modal consistency serves as an effective self-supervisory signal, improving the performance of individual sensors by forcing them to "agree" on object identities.

  • Accuracy Improvements: On the KITTI-360 benchmark, the addition of temporal RGB consistency improved mean Average Precision (mAP) by 1.9 points, while cross-modal consistency (LiDAR + RGB) added another 1.7 mAP.
  • Temporal Reliability: The method significantly enhanced temporal stability, reducing embedding variance over time and improving temporal accuracy from 75.1% to 83.4%.
  • Robustness to Rare Classes: The model showed marked improvements in segmenting safety-critical but less frequent classes, such as pedestrians and cyclists.
Qualitative visualizations show the model's ability to disentangle individual cars and pedestrians in crowded urban environments.

Conclusion

By enforcing consistency across sensors and time, this approach creates a more reliable and accurate foundation for instance segmentation in autonomous systems. The learned embeddings are not only more accurate but also more stable, providing a "stickier" identity for objects as they move through a scene. Future work will explore replacing the simple clustering post-processing with more advanced architectures like Transformers to further push the boundaries of segmentation performance.

Summary updated

BibTeX
@inproceedings{zhu2022instance,
  author = {Alex Zihao Zhu and Vincent Casser and Reza Mahjourian and Henrik Kretzschmar and Sören Pirk},
  title = {{Instance Segmentation with Cross-Modal Consistency}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=6b5M4naiXi3},
}
Figure