TL;DR
This paper introduces Cross-modality Noisy Supervision (CNS), a framework that enables 2D and 3D neural networks to achieve semantic scene understanding without a single human-annotated label. By strategically combining the semantic knowledge of CLIP with the precise spatial masks of the Segment Anything Model (SAM), the authors achieve state-of-the-art "label-free" performance on both indoor and outdoor benchmarks.
What Problem Does This Paper Address?
Traditional scene understanding—identifying what and where objects are in a 3D environment—is a data-hungry process. Training these models typically requires massive amounts of manually labeled 2D images and 3D point clouds. This is not only expensive and time-consuming but also limits models to a "closed-set" vocabulary; they often fail when encountering novel objects not present in their training data.
Recent "vision foundation models" like CLIP (Contrastive Vision-Language Pre-training) and SAM (Segment Anything Model) offer a way out. CLIP understands a vast array of semantic concepts from the web, and SAM can segment almost any object mask with high precision. However, using them for label-free training is difficult: CLIP’s pixel-level predictions are notoriously noisy, and SAM provides masks but has no idea what those masks actually represent. Furthermore, projecting 2D labels into 3D space introduces additional calibration errors, resulting in "extremely noisy" pseudo-labels that can easily lead a network to overfit on garbage data.
Methods
The authors propose the Cross-modality Noisy Supervision (CNS) framework to turn these noisy foundation model outputs into reliable training signals. The architecture is illustrated in the figure below:
Key components include:
- Label Refinement via SAM: The system starts with CLIP-generated pseudo-labels, which are often fragmented. By applying a majority-voting strategy within each SAM-generated mask, the labels are "cleaned," ensuring that all pixels within a single object mask share the same semantic identity.
- Prediction Consistency Regularization: To prevent the 2D and 3D networks from simply memorizing noise, the authors use a co-training strategy. During training, pseudo-labels are randomly switched between the 2D network, the 3D network, and the refined CLIP outputs. This forces the networks to learn robust features that are consistent across modalities.
- Latent Space Consistency: The authors leverage SAM’s robust feature representation as an "anchor." They regularize the latent spaces of both the 2D and 3D networks to align with the frozen SAM feature space, which helps the networks produce semantic predictions with much sharper boundaries.
Datasets
The researchers evaluated their method on three major benchmarks:
- ScanNet: A large-scale indoor dataset consisting of 1,603 RGB-D scans of various rooms.
- nuScenes: A popular outdoor autonomous driving dataset featuring LiDAR scans and multi-view camera images.
- nuImages: A 2D companion to nuScenes providing annotated images for outdoor scenes.
Key Insights and Ideas
- The Semantic-Spatial Synergy: The core "aha moment" is that CLIP and SAM are perfectly complementary for this task. CLIP provides the what (semantics) but lacks the where (spatial precision), while SAM provides the where but lacks the what. CNS is the bridge that unites them.
- Noise as a Regularizer: Instead of trying to perfectly eliminate label noise, the framework embraces it. By randomly switching labels between modalities during co-training, the model is prevented from converging on the specific errors of any single source (like 2D-to-3D projection errors).
- SAM as a Metric Anchor: Using SAM's feature space as a frozen reference point for latent space alignment is a clever way to transfer SAM's "segment anything" capability into the target networks without needing explicit labels.
Key Results
The CNS framework significantly outperformed previous state-of-the-art label-free methods like CLIP2Scene and MaskCLIP.
- ScanNet (Indoor): The 3D network achieved 33.5% mIoU, a substantial 7.9% improvement over previous methods. The 2D network reached 28.4% mIoU.
- nuScenes (Outdoor): The method achieved 26.8% mIoU for 3D segmentation, marking a 6.0% improvement.
Qualitative results (see Figure 3) show that the model produces segments that are remarkably close to human-annotated ground truth, even for complex indoor scenes.
Limitations and Future Work
The primary limitation is the framework's reliance on CLIP’s internal vocabulary. If a concept is rare or absent in CLIP’s training data—such as the specific term "derivable surface" in driving contexts—the model struggles to label it correctly. The authors suggest that future work could involve more sophisticated prompt engineering or exploring alternative foundation models to generate even more descriptive text embeddings. Additionally, while the method is label-free, it still requires calibrated 2D and 3D data to perform the initial projections.
Summary updated