Towards Granularity-adjusted Pixel-level Semantic Annotation

Rohit Kundu, Sudipta Paul, Rohit Lal, Amit K. Roy-Chowdhury

TL;DR

This paper introduces U-SAM, a novel framework that imbues the powerful Segment Anything Model (SAM) with semantic awareness. U-SAM allows users to generate precise, category-specific pixel-level segmentation masks from unlabeled images, simply by providing object class names, eliminating the need for manual annotation or dataset-specific training.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task for understanding images in applications like autonomous driving and medical imaging, traditionally relies on vast amounts of meticulously labeled data. This manual annotation process is incredibly time-consuming and expensive, creating a significant bottleneck. While models like SAM can generate highly accurate object masks, they lack the ability to identify what those objects are, outputting generic, unlabeled masks. This paper addresses the gap of generating semantically aware pixel-level annotations on unlabeled data, without requiring any prior knowledge of the test dataset's distribution or manual supervision.

Methods

The core of U-SAM is to enhance SAM's existing capabilities. The process involves:

  1. Leveraging SAM's Mask Generation: U-SAM utilizes SAM's "automatic" mode to generate a multitude of masks for any given image, covering various granularities.
  2. Acquiring Semantic Information: To imbue semantic understanding, U-SAM uses either synthetically generated images (from models like Stable Diffusion) or web-crawled images that correspond to the user-specified object classes. This data is used to learn the semantic meaning of the desired objects.
  3. Learning a Mapping: A classifier head is trained on top of SAM's mask embeddings. This classifier learns to map the mask embeddings to the corresponding object class labels. This is achieved through a weakly-supervised approach using Multiple Instance Learning (MIL) and an uncertainty distillation technique to improve robustness.
  4. Inference: During inference, SAM generates masks, and the trained classifier then assigns semantic labels to these masks. Post-processing steps like Non-Maximal Suppression (NMS) are used to refine the final output and ensure distinct object masks.

Datasets

  • Training Data:
    • Synthetic images generated by Stable Diffusion.
  • Web-crawled images.
  • Evaluation Datasets:
    • PASCAL VOC 2012
    • MSCOCO-80

The paper highlights that the training data (synthetic or web-crawled) is often single-object and distinct from the test data distribution, emphasizing U-SAM's generalization capabilities.

Key Insights and Ideas

  • SAM's Semantic Blind Spot: The paper clearly articulates that SAM, while excellent at what to segment, fails at what the segmented object is. This is a critical limitation for practical applications.
  • Bridging the Gap with External Data: The ingenious use of synthetic or web-crawled data, guided by user-provided class names, provides a scalable way to teach SAM semantics without needing labeled test data.
  • Weak Supervision is Sufficient: By framing the problem as learning from "bags" of masks (MIL), the approach bypasses the need for precise pixel-level labels during the semantic learning phase.
  • Uncertainty Matters: The incorporation of uncertainty distillation helps the model become more robust, especially in challenging scenarios or when dealing with diverse data.

Key Results

U-SAM demonstrates significant improvements over state-of-the-art methods:

  • PASCAL VOC 2012: Achieved +17.95% mIoU improvement over unsupervised methods and +21.67% over weakly supervised methods when trained on synthetic/web-crawled data.
  • MSCOCO-80: Achieved +5.20% mIoU improvement over unsupervised methods and +7.99% over weakly supervised methods when trained on synthetic/web-crawled data.
  • Qualitative results show U-SAM producing targeted masks for user-specified objects, unlike SAM's generic outputs, and even correctly identifying objects missed in ground truth annotations.

Limitations and Future Work

  • Granularity Control: While U-SAM aims for user-defined granularity, the paper notes that SAM itself produces masks at various levels. The current approach focuses on selecting masks relevant to the user's specified classes, implying that fine-grained control over the exact level of detail might still be an area for further exploration.
  • Reliance on External Data Quality: The performance of U-SAM is dependent on the quality and representativeness of the synthetic or web-crawled data used for training.
  • Future Work: The paper suggests that U-SAM's architecture can be extended to handle more complex scenarios and potentially adapt to new domains with minimal effort, hinting at future research in domain generalization and adaptation. The paper also explores changing granularity definitions for classes, suggesting flexibility in how users can define their segmentation tasks.

Summary updated

Abstract

Recent advancements in computer vision predominantly rely on learning-based systems, leveraging annotations as the driving force to develop specialized models. However, annotating pixel-level information, particularly in semantic segmentation, presents a challenging and labor-intensive task, prompting the need for autonomous processes. In this work, we propose GranSAM which distinguishes itself by providing semantic segmentation at the user-defined granularity level on unlabeled data without the need for any manual supervision, offering a unique contribution in the realm of semantic mask annotation method. Specifically, we propose an approach to enable the Segment Anything Model (SAM) with semantic recognition capability to generate pixel-level annotations for images without any manual supervision. For this, we accumulate semantic information from synthetic images generated by the Stable Diffusion model or web crawled images and employ this data to learn a mapping function between SAM mask embeddings and object class labels. As a result, SAM, enabled with granularity-adjusted mask recognition, can be used for pixel-level semantic annotation purposes. We conducted experiments on the PASCAL VOC 2012 and COCO-80 datasets and observed a +17.95% and +5.17% increase in mIoU, respectively, compared to existing state-of-the-art methods when evaluated under our problem setting.
BibTeX
@misc{kundu2023towards,
  author = {Rohit Kundu and Sudipta Paul and Rohit Lal and Amit K. Roy-Chowdhury},
  title = {{Towards Granularity-adjusted Pixel-level Semantic Annotation}},
  year = {2023},
  eprint = {2312.02420},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.02420},
}
Figure