Abstract
3D semantic segmentation is a critical task for applications in robotics and augmented reality, yet it typically requires labor-intensive point-level annotations. Weakly supervised methods using scene-level labels offer a more scalable alternative but often struggle to integrate geometric and appearance information or handle the long-tailed distribution of point cloud data. This paper presents a Multi-modality Affinity (MMA) inference framework that leverages both 3D geometry and RGB color data to improve point affinity learning. By incorporating Normalizing Classifier Weights (NCW) and a teacher-student transformer architecture, our method achieves state-of-the-art performance on major benchmarks, outperforming previous scene-level supervised methods by significant margins.
Introduction
Labeling millions of points in a 3D scan is a bottleneck for developing robust semantic segmentation models. Scene-level weak supervision—where a model is only told which objects are present in a room without their locations—drastically reduces annotation costs. However, current weakly supervised approaches often ignore the rich interplay between shape (geometry) and texture (color). Furthermore, they are frequently biased toward "head" classes (like walls and floors) while failing to recognize "tail" classes (like sinks or toilets). This paper addresses these gaps through a multi-modal approach that refines point affinity and balances class representation.
Methodology: The MMA Framework
The proposed framework, Multi-modality Affinity (MMA), utilizes a two-stream architecture to process geometric and appearance data independently before fusing them for affinity inference.
Multi-modality Affinity Inference
At the heart of the system is the Affinity Inference Module. By processing both masked RGB (pure geometry) and RGB-appended inputs, the model learns to identify objects based on both their physical structure and their visual appearance. This dual-stream approach ensures that the model remains robust even when one modality is ambiguous (e.g., a floor with complex shadows).
Long-tail Aware Enhancement (NCW)
To combat the long-tail problem, the authors introduce Normalizing Classifier Weights (NCW). In standard models, weights for common classes tend to have larger magnitudes, biasing the affinity scores. By normalizing these weights to a uniform L2 norm, the model produces more balanced affinity maps, allowing rare objects to be identified as distinct entities rather than being absorbed into common background classes.
Teacher-Student Transformer
The segmentation module employs a teacher-student setup. The "Teacher" head generates pseudo-labels using Multiple Instance Learning (MIL), which then guide the "Student" head. The student utilizes a Transformer layer to capture long-range spatial dependencies, helping the model understand that disparate parts of a large object (like a long table) belong to the same semantic category.
Experimental Results
The MMA method was evaluated on the ScanNet and S3DIS benchmarks, setting new records for scene-level weakly supervised 3D segmentation.
- ScanNet: The model achieved 30.6% mIoU on the test set, a 6.6% improvement over the previous state-of-the-art (WyPR).
- S3DIS: The model reached 26.3% mIoU, surpassing the previous best by 4.0%.
Qualitative visualizations demonstrate that MMA produces significantly sharper object boundaries. While baseline models often "bleed" labels across adjacent surfaces, MMA correctly distinguishes between distinct objects like doors and walls or desks and chairs, even with minimal supervision.
Conclusion
By effectively combining multi-modal features and addressing class imbalance through weight normalization, the MMA framework significantly closes the gap between weakly supervised and fully supervised 3D segmentation. This approach provides a robust path forward for training high-performance 3D models with minimal human annotation effort.
Summary updated