Towards Label-free Scene Understanding by Vision Foundation Models

Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, Wenping Wang

TL;DR

This paper introduces Cross-modality Noisy Supervision (CNS), a framework that enables 2D and 3D neural networks to achieve semantic scene understanding without a single human-annotated label. By strategically combining the semantic knowledge of CLIP with the precise spatial masks of the Segment Anything Model (SAM), the authors achieve state-of-the-art "label-free" performance on both indoor and outdoor benchmarks.

What Problem Does This Paper Address?

Traditional scene understanding—identifying what and where objects are in a 3D environment—is a data-hungry process. Training these models typically requires massive amounts of manually labeled 2D images and 3D point clouds. This is not only expensive and time-consuming but also limits models to a "closed-set" vocabulary; they often fail when encountering novel objects not present in their training data.

Recent "vision foundation models" like CLIP (Contrastive Vision-Language Pre-training) and SAM (Segment Anything Model) offer a way out. CLIP understands a vast array of semantic concepts from the web, and SAM can segment almost any object mask with high precision. However, using them for label-free training is difficult: CLIP’s pixel-level predictions are notoriously noisy, and SAM provides masks but has no idea what those masks actually represent. Furthermore, projecting 2D labels into 3D space introduces additional calibration errors, resulting in "extremely noisy" pseudo-labels that can easily lead a network to overfit on garbage data.

Methods

The authors propose the Cross-modality Noisy Supervision (CNS) framework to turn these noisy foundation model outputs into reliable training signals. The architecture is illustrated in the figure below:

Figure 2: Illustration of the overall framework. Specifically, we employ CLIP to densely pseudo-label 2D image pixels and transfer these labels to 3D points using the calibration matrix. Acknowledging that network predictions and pseudo-labels are extremely noisy, we refine pseudo-labels using SAM’s masks for more reliable supervision. To mitigate error propagation and prevent the networks from overfitting the noisy labels, we randomly switch the different refined pseudo-labels to co-training the 2D and 3D networks. Besides, we consistently regularize the network’s latent space using SAM’s robust feature, which benefits the networks to produce clearer predictions.

Key components include:

  • Label Refinement via SAM: The system starts with CLIP-generated pseudo-labels, which are often fragmented. By applying a majority-voting strategy within each SAM-generated mask, the labels are "cleaned," ensuring that all pixels within a single object mask share the same semantic identity.
  • Prediction Consistency Regularization: To prevent the 2D and 3D networks from simply memorizing noise, the authors use a co-training strategy. During training, pseudo-labels are randomly switched between the 2D network, the 3D network, and the refined CLIP outputs. This forces the networks to learn robust features that are consistent across modalities.
  • Latent Space Consistency: The authors leverage SAM’s robust feature representation as an "anchor." They regularize the latent spaces of both the 2D and 3D networks to align with the frozen SAM feature space, which helps the networks produce semantic predictions with much sharper boundaries.

Datasets

The researchers evaluated their method on three major benchmarks:

  1. ScanNet: A large-scale indoor dataset consisting of 1,603 RGB-D scans of various rooms.
  2. nuScenes: A popular outdoor autonomous driving dataset featuring LiDAR scans and multi-view camera images.
  3. nuImages: A 2D companion to nuScenes providing annotated images for outdoor scenes.

Key Insights and Ideas

  • The Semantic-Spatial Synergy: The core "aha moment" is that CLIP and SAM are perfectly complementary for this task. CLIP provides the what (semantics) but lacks the where (spatial precision), while SAM provides the where but lacks the what. CNS is the bridge that unites them.
  • Noise as a Regularizer: Instead of trying to perfectly eliminate label noise, the framework embraces it. By randomly switching labels between modalities during co-training, the model is prevented from converging on the specific errors of any single source (like 2D-to-3D projection errors).
  • SAM as a Metric Anchor: Using SAM's feature space as a frozen reference point for latent space alignment is a clever way to transfer SAM's "segment anything" capability into the target networks without needing explicit labels.

Key Results

The CNS framework significantly outperformed previous state-of-the-art label-free methods like CLIP2Scene and MaskCLIP.

  • ScanNet (Indoor): The 3D network achieved 33.5% mIoU, a substantial 7.9% improvement over previous methods. The 2D network reached 28.4% mIoU.
  • nuScenes (Outdoor): The method achieved 26.8% mIoU for 3D segmentation, marking a 6.0% improvement.

Qualitative results (see Figure 3) show that the model produces segments that are remarkably close to human-annotated ground truth, even for complex indoor scenes.

Figure 3: Qualitative results of label-free semantic segmentation on the ScanNet 25k images dataset. From the left to the right column are the input, prediction by SAM, CLIP, ours w/o CNS, our full method, and ground truth, respectively.

Limitations and Future Work

The primary limitation is the framework's reliance on CLIP’s internal vocabulary. If a concept is rare or absent in CLIP’s training data—such as the specific term "derivable surface" in driving contexts—the model struggles to label it correctly. The authors suggest that future work could involve more sophisticated prompt engineering or exploring alternative foundation models to generate even more descriptive text embeddings. Additionally, while the method is label-free, it still requires calibrated 2D and 3D data to perform the initial projections.

Summary updated

Abstract

Vision foundation models such as Contrastive Vision-Language Pre-training (CLIP) and Segment Anything (SAM) have demonstrated impressive zero-shot performance on image classification and segmentation tasks. However, the incorporation of CLIP and SAM for label-free scene understanding has yet to be explored. In this paper, we investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. The primary challenge lies in effectively supervising networks under extremely noisy pseudo labels, which are generated by CLIP and further exacerbated during the propagation from the 2D to the 3D domain. To tackle these challenges, we propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. In particular, we introduce a prediction consistency regularization to co-train 2D and 3D networks, then further impose the networks' latent space consistency using the SAM's robust feature representation. Experiments conducted on diverse indoor and outdoor datasets demonstrate the superior performance of our method in understanding 2D and 3D open environments. Our 2D and 3D network achieves label-free semantic segmentation with 28.4\% and 33.5\% mIoU on ScanNet, improving 4.7\% and 7.9\%, respectively. For nuImages and nuScenes datasets, the performance is 22.1\% and 26.8\% with improvements of 3.5\% and 6.0\%, respectively. Code is available. (https://github.com/runnanchen/Label-Free-Scene-Understanding).
BibTeX
@inproceedings{chen2023towards,
  author = {Runnan Chen and Youquan Liu and Lingdong Kong and Nenglun Chen and Xinge Zhu and Yuexin Ma and Tongliang Liu and Wenping Wang},
  title = {{Towards Label-free Scene Understanding by Vision Foundation Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=C8JdyM7B8I},
}
Figure