Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

Xiawei Li, Qingyuan Xu, Jing Zhang, Tianyi Zhang, Qian Yu, Lu Sheng, Dong Xu

Abstract

3D semantic segmentation is a critical task for applications in robotics and augmented reality, yet it typically requires labor-intensive point-level annotations. Weakly supervised methods using scene-level labels offer a more scalable alternative but often struggle to integrate geometric and appearance information or handle the long-tailed distribution of point cloud data. This paper presents a Multi-modality Affinity (MMA) inference framework that leverages both 3D geometry and RGB color data to improve point affinity learning. By incorporating Normalizing Classifier Weights (NCW) and a teacher-student transformer architecture, our method achieves state-of-the-art performance on major benchmarks, outperforming previous scene-level supervised methods by significant margins.

Introduction

Labeling millions of points in a 3D scan is a bottleneck for developing robust semantic segmentation models. Scene-level weak supervision—where a model is only told which objects are present in a room without their locations—drastically reduces annotation costs. However, current weakly supervised approaches often ignore the rich interplay between shape (geometry) and texture (color). Furthermore, they are frequently biased toward "head" classes (like walls and floors) while failing to recognize "tail" classes (like sinks or toilets). This paper addresses these gaps through a multi-modal approach that refines point affinity and balances class representation.

Methodology: The MMA Framework

The proposed framework, Multi-modality Affinity (MMA), utilizes a two-stream architecture to process geometric and appearance data independently before fusing them for affinity inference.

Overview of the MMA framework architecture showing feature extraction, affinity inference, and segmentation modules.

Multi-modality Affinity Inference

At the heart of the system is the Affinity Inference Module. By processing both masked RGB (pure geometry) and RGB-appended inputs, the model learns to identify objects based on both their physical structure and their visual appearance. This dual-stream approach ensures that the model remains robust even when one modality is ambiguous (e.g., a floor with complex shadows).

Long-tail Aware Enhancement (NCW)

To combat the long-tail problem, the authors introduce Normalizing Classifier Weights (NCW). In standard models, weights for common classes tend to have larger magnitudes, biasing the affinity scores. By normalizing these weights to a uniform L2 norm, the model produces more balanced affinity maps, allowing rare objects to be identified as distinct entities rather than being absorbed into common background classes.

Teacher-Student Transformer

The segmentation module employs a teacher-student setup. The "Teacher" head generates pseudo-labels using Multiple Instance Learning (MIL), which then guide the "Student" head. The student utilizes a Transformer layer to capture long-range spatial dependencies, helping the model understand that disparate parts of a large object (like a long table) belong to the same semantic category.

Experimental Results

The MMA method was evaluated on the ScanNet and S3DIS benchmarks, setting new records for scene-level weakly supervised 3D segmentation.

  • ScanNet: The model achieved 30.6% mIoU on the test set, a 6.6% improvement over the previous state-of-the-art (WyPR).
  • S3DIS: The model reached 26.3% mIoU, surpassing the previous best by 4.0%.
Qualitative comparison of segmentation results on ScanNet, showing MMA's ability to produce cleaner boundaries and more accurate object identification compared to baselines.

Qualitative visualizations demonstrate that MMA produces significantly sharper object boundaries. While baseline models often "bleed" labels across adjacent surfaces, MMA correctly distinguishes between distinct objects like doors and walls or desks and chairs, even with minimal supervision.

Conclusion

By effectively combining multi-modal features and addressing class imbalance through weight normalization, the MMA framework significantly closes the gap between weakly supervised and fully supervised 3D segmentation. This approach provides a robust path forward for training high-performance 3D models with minimal human annotation effort.

Summary updated

Abstract

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.
BibTeX
@inproceedings{li2024multi,
  author = {Xiawei Li and Qingyuan Xu and Jing Zhang and Tianyi Zhang and Qian Yu and Lu Sheng and Dong Xu},
  title = {{Multi-modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28106},
}
Figure