Introduction
This paper introduces a novel joint framework for 2D and 3D weakly supervised semantic segmentation (WSSS) that relies entirely on scene-level class labels. By creating a bidirectional self-supervision loop—where 2D "semantic richness" guides 3D perception and 3D "geometric precision" sharpens 2D boundaries—the authors achieve state-of-the-art results on major benchmarks while significantly reducing the need for expensive manual annotations.
The Challenge of Isolated WSSS
Semantic segmentation, the task of labeling every pixel in an image or every point in a 3D cloud, is a cornerstone of computer vision. However, training these models typically requires "dense" labels, which are incredibly labor-intensive to produce (e.g., it takes over 20 minutes to manually annotate a single 3D point cloud).
Weakly Supervised Semantic Segmentation (WSSS) attempts to solve this by training models using "cheap" labels, like image-level tags (e.g., "this image contains a chair"). The challenge is that 2D and 3D WSSS have distinct, domain-specific weaknesses:
- 2D WSSS is highly effective at identifying objects but struggles to find precise boundaries, often producing "blurry" or bleeding segmentation masks.
- 3D WSSS understands physical geometry well but suffers from severe class imbalance (e.g., floors and walls dominate the scene, while small objects are rare) and sparse data.
Existing research usually treats these as isolated problems. This paper addresses the missed opportunity of using 2D and 3D data together to cancel out each domain's respective flaws.
A Bidirectional Joint Framework
The authors propose a Joint 2D-3D WSSS framework that utilizes a bidirectional transfer of knowledge during the training phase:
- 2D-to-3D (Transferring Semantics): The model generates Class Activation Maps (CAMs) from 2D images, which highlight the pixels that "triggered" the classifier. These 2D maps are then projected into 3D space to act as pseudo-labels. This helps the 3D network recognize objects it might otherwise miss due to point cloud sparsity.
- 3D-to-2D (Transferring Geometry): To fix the "blurry boundary" problem in 2D, the authors leverage 3D features. They calculate a correlation matrix of 3D points to determine which points are geometrically similar. They then force the 2D image classifier to produce features with a similar correlation structure. This effectively "teaches" the 2D model to respect the sharp physical boundaries found in the 3D geometry.
- Confidence-Based Scoring: Because pseudo-labels are inherently noisy, the authors developed a confidence-based scoring method to weight the training loss. If the model is unsure about a specific region, that region contributes less to the training updates, preventing the two networks from "learning" each other's mistakes.
Crucially, while the models are trained jointly, they can be used individually during the inference phase. A 3D scan is not required to run the trained 2D segmenter, and vice versa.
Key Insights
- Complementary Strengths: The core insight is that 2D and 3D data are complementary, not just redundant. 2D provides the what (texture and color semantics), while 3D provides the where (structural geometry).
- Geometry as a Regularizer: Using 3D feature correlations to guide 2D learning is a highly effective way to inject "physical common sense" into an image-based model without needing explicit 2D boundary labels.
- Data Efficiency: The authors achieved their results using only 1% of the available RGB frames from the dataset (about 17 frames per scene), proving that the method is highly efficient.
Key Results
Tested primarily on the ScanNetV2 benchmark (1,513 indoor scans) and S3DIS (for generalization), the joint framework demonstrated remarkable improvements.
- State-of-the-Art Performance: On the ScanNetV2 benchmark, the joint framework achieved a mean Intersection over Union (mIoU) of 49.6% on the validation set and 47.4% on the test set.
- Significant Gains: This performance outperforms previous 3D WSSS leaders (such as MPRM and WyPR) by more than 6%.
- Ablation Success: The bidirectional loop is vital. Without the joint learning components, the 3D model's performance was a meager 16.2%. Adding the 2D-to-3D transfer alone boosted it to 43.8%, and the full bidirectional loop brought it to nearly 50%.
Limitations and Future Directions
While highly effective, the approach has a few limitations:
- Field of View (FoV) Mismatch: Standard cameras have a limited FoV. If an object is only partially visible in an image but fully present in the 3D scan, the 2D-to-3D transfer can become inaccurate. Future work may explore omnidirectional cameras to mitigate this.
- Paired Data Requirement: Although the models can run separately after training, the training process itself strictly requires paired 2D and 3D data, which may not be available for legacy datasets.
- Inference Noise: The model handles training noise well via confidence scoring, but misclassifications during the inference phase remain a challenge that the current framework does not fully resolve.
Summary updated