Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation

Xiawei Li, Qingyuan Xu, Jing Zhang, Tianyi Zhang, Qian Yu, Lu Sheng, Dong Xu

Abstract

Weakly supervised 3D semantic segmentation (WSSS) aims to reduce the prohibitive cost of point-level annotations by learning from coarser labels. This paper addresses the limitations of current scene-level WSSS methods, which often neglect the synergy between geometric and appearance modalities and suffer from long-tail distribution biases. We propose the Multi-Modality Affinity Inference (MMA) framework, which utilizes a two-stream architecture to fuse geometric and RGB features for robust point affinity estimation. Additionally, we introduce Normalizing Classifier Weights (NCW) to mitigate bias toward common classes. Experimental results on ScanNet and S3DIS benchmarks demonstrate that MMA significantly outperforms existing state-of-the-art methods, achieving up to an 8.1% mIoU improvement on ScanNet.

Introduction

3D semantic segmentation is a foundational task for applications in robotics, autonomous driving, and augmented reality. While deep learning has driven significant progress, the reliance on dense, point-level annotations remains a major bottleneck. Annotating a single 3D scene can take hours of manual labor, making large-scale dataset creation expensive and slow.

Weakly Supervised Semantic Segmentation (WSSS) using scene-level labels—where the model is only informed of which objects are present in a scene—offers a cost-effective alternative. However, the lack of spatial supervision makes it difficult to delineate precise object boundaries. Current methods often fail to leverage the full potential of multi-modal data (RGB-D) and struggle with the "long-tail" problem, where dominant classes like walls and floors overshadow rare objects like toilets or bathtubs.

Methodology

Multi-Modality Affinity Inference (MMA)

The core of our approach is the Multi-Modality Affinity Inference (MMA) module. Unlike previous methods that rely primarily on one data type, MMA processes 3D scans through two parallel streams:

  1. Geometric Stream: Processes masked point clouds containing only 3D coordinates to capture pure structural and scale information.
  2. RGB-Appended Stream: Utilizes full RGB-D data to incorporate texture and color cues.
Overview of the Multi-Modality Affinity Inference (MMA) framework, featuring two-stream feature extraction and affinity inference modules.

By comparing features from both streams, the model learns a comprehensive affinity matrix. This matrix defines the similarity between points, allowing the model to "propagate" scene-level labels across geometrically and visually consistent regions. The use of the Z-axis (height) in the geometric stream is particularly effective at distinguishing objects that might share similar 2D textures but occupy different 3D spaces, such as a rug versus a floor.

Normalizing Classifier Weights (NCW)

To address the long-tail problem, we introduce Normalizing Classifier Weights (NCW). Standard classifiers often assign larger weight magnitudes to frequent "head" classes. By enforcing an L2-norm normalization on the classifier weights, we ensure that all classes—regardless of their frequency in the training set—have an equal influence on the feature space. This prevents the model from becoming biased toward common structures and improves the recognition of rare "tail" objects.

Teacher-Student Learning

The framework employs a Teacher-Student paradigm. The Teacher network generates initial pseudo-labels using Multiple Instance Learning (MIL). A Student network, enhanced with a Transformer layer to capture long-range spatial relationships, is then trained to refine these labels, leading to more coherent and accurate final segmentations.

Experimental Evaluation

Quantitative Results

We evaluated MMA on two major indoor 3D datasets: ScanNet and S3DIS.

  • ScanNet: MMA achieved 37.7% mIoU, surpassing the previous state-of-the-art (WyPR) by 8.1%.
  • S3DIS: The model reached 26.3% mIoU, a 4.0% improvement over existing baselines.
  • Long-Tail Performance: The NCW module led to a 7.2% increase in mean Average Precision (mAP) for rare classes, demonstrating its effectiveness in balancing the model's focus.

Qualitative Analysis

Visual comparisons highlight the precision of the MMA framework. While baseline models often produce noisy boundaries or fail to detect small objects, MMA generates clean, well-defined segments that closely match the ground truth.

Qualitative comparison of 3D semantic segmentation results on ScanNet and S3DIS datasets, demonstrating the improvement from baseline to the final MMA model.

Conclusion

The Multi-Modality Affinity Inference framework represents a significant step forward for weakly supervised 3D segmentation. By effectively fusing geometric and appearance data and addressing class imbalance through weight normalization, MMA achieves superior performance without the need for complex pre-processing or expensive point-level labels. Future work will focus on improving cross-scene generalization and reducing the computational overhead associated with multi-stream training.

Summary updated

Abstract

3D point cloud semantic segmentation has a wide range of applications. Recently, weakly supervised point cloud segmentation methods have been proposed, aiming to alleviate the expensive and laborious manual annotation process by leveraging scene-level labels. However, these methods have not effectively exploited the rich geometric information (such as shape and scale) and appearance information (such as color and texture) present in RGB-D scans. Furthermore, current approaches fail to fully leverage the point affinity that can be inferred from the feature extraction network, which is crucial for learning from weak scene-level labels. Additionally, previous work overlooks the detrimental effects of the long-tailed distribution of point cloud data in weakly supervised 3D semantic segmentation. To this end, this paper proposes a simple yet effective scene-level weakly supervised point cloud segmentation method with a newly introduced multi-modality point affinity inference module. The point affinity proposed in this paper is characterized by features from multiple modalities (e.g., point cloud and RGB), and is further refined by normalizing the classifier weights to alleviate the detrimental effects of long-tailed distribution without the need of the prior of category distribution. Extensive experiments on the ScanNet and S3DIS benchmarks verify the effectiveness of our proposed method, which outperforms the state-of-the-art by ~4% to ~ 6% mIoU. Codes are released at https://github.com/Sunny599/AAAI24-3DWSSG-MMA.
BibTeX
@inproceedings{li2024multi,
  author = {Xiawei Li and Qingyuan Xu and Jing Zhang and Tianyi Zhang and Qian Yu and Lu Sheng and Dong Xu},
  title = {{Multi-Modality Affinity Inference for Weakly Supervised 3D Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28106},
}
Figure