Abstract
Manual annotation of every point in a point cloud is a costly and labor-intensive process. While weakly supervised point cloud semantic segmentation (WSPCSS) with sparse annotation shows promise, the limited information from initial sparse labels can place an upper bound on performance. As a new research direction for WSPCSS, we propose a novel Region Exploration via Artificial Labeling (REAL) framework. It leverages a foundational image model as an artificial oracle within the active learning context, eliminating the need for manual annotation by a human oracle. To integrate the 2D model into the 3D domain, we first introduce a Projection-based Point-to-Segment (PP2S) module, designed to enable prompt segmentation of 3D data without additional training. The REAL framework samples query points based on model predictions and requests annotations from PP2S, dynamically refining labels and improving model training. Furthermore, to overcome several challenges of employing an artificial model as an oracle, we formulate effective query sampling and label updating strategies. Our comprehensive experiments and comparisons demonstrate that the REAL framework significantly outperforms existing methods across various benchmarks.
Introduction
Semantic segmentation—the task of assigning a category (like "chair" or "wall") to every point in a 3D scan—is essential for robotics and autonomous systems. However, 3D point clouds are massive, and manually labeling every single point is prohibitively expensive. Weakly Supervised Point Cloud Semantic Segmentation (WSPCSS) addresses this by using only a tiny fraction of labels (e.g., 0.02%).
Despite its promise, WSPCSS faces a "performance ceiling" because sparse labels provide insufficient geometric and boundary information. Most existing methods attempt to maximize the utility of these few labels. This paper introduces a different approach: generating high-quality labels autonomously using a 2D foundation model as an "Artificial Oracle."
The REAL Framework
The Region Exploration via Artificial Labeling (REAL) framework treats a 2D foundation model, specifically the Segment Anything Model (SAM), as an automated teacher within an active learning loop.
Bridging the Dimensional Gap: PP2S
To utilize 2D models for 3D data, the researchers developed the Projection-based Point-to-Segment (PP2S) module. Since SAM operates on 2D images, PP2S projects 3D points onto corresponding 2D camera views. SAM then generates high-quality 2D masks around these points, which are back-projected into 3D space. A multi-view voting system ensures that labels remain consistent across different camera angles, effectively transferring 2D structural knowledge into the 3D domain.
Cautious Label Updating
Because 3D-to-2D projection is susceptible to calibration errors and occlusions, the framework employs a "cautious" update strategy. The system only accepts a new label from the Artificial Oracle if the oracle’s segment aligns with the 3D model’s current prediction. This filtering mechanism prevents noisy labels from degrading the training process.
Experimental Results
The REAL framework was evaluated on the S3DIS and ScanNetV2 benchmarks under various sparsity settings, including an extreme "1 point per object" scenario (approximately 0.004% of total points).
Key Findings
- Efficiency: On the S3DIS dataset, REAL achieved 62.7% mIoU using only 0.004% labels, outperforming previous state-of-the-art (SoTA) methods that required five times more human effort (0.02% labels).
- Performance: Under standard 0.02% supervision, REAL reached 65.2% mIoU, a significant improvement over the previous best of 62.3%.
- Visual Quality: Qualitative results show that the framework produces cleaner object boundaries and handles complex, cluttered indoor scenes more effectively than baseline models.
Conclusion
The REAL framework demonstrates that 2D foundation models can serve as effective "Artificial Oracles" for 3D tasks. By bridging the gap between 2D imagery and 3D point clouds, the system achieves superior segmentation performance while drastically reducing the need for manual human annotation. Future work may involve integrating semantic-aware foundation models to further reduce the reliance on initial human-provided class names.
Summary updated