Semantic-Aware Superpixel for Weakly Supervised Semantic Segmentation

Sangtae Kim, Daeyoung Park, Byonghyo ShimAAAI 2023 · DOI

Abstract

This paper introduces a novel approach to Weakly Supervised Semantic Segmentation (WSSS) that eliminates the need for pixel-level saliency detectors. By leveraging features from a self-supervised Vision Transformer (DINO), the authors generate "semantic-aware superpixels" that capture object boundaries and global context. These superpixels guide a seeded region-growing process to train a high-performing segmentation network using only image-level labels.

Introduction

Semantic segmentation—assigning a class label to every pixel in an image—is a cornerstone of computer vision but typically requires massive datasets with dense, pixel-level annotations. Creating these is incredibly laborious. Weakly Supervised Semantic Segmentation (WSSS) attempts to train models using only "weak" labels, such as image-level tags (e.g., "this image contains a cat").

The standard tool for WSSS is the Class Activation Map (CAM), which identifies which pixels a classification network focuses on. However, CAMs are notoriously "blotchy," highlighting only the most discriminative parts of an object while missing boundaries and the full extent of the body. To address this, most state-of-the-art methods rely on external saliency detectors. However, these detectors often require pixel-level labels for their own training, which partially defeats the purpose of WSSS. This paper proposes a method to achieve high-quality boundaries using purely unsupervised features.

Methodology

The authors propose a two-stage framework that leverages self-supervised features to guide segmentation.

Semantic-Aware Superpixel Discovery

Instead of traditional color-based superpixels (like SLIC), the method uses features from DINO, a Vision Transformer trained via self-distillation without labels. DINO features are known to contain explicit information about object shapes. The process involves:

  • Similarity Matrix: Calculating a similarity matrix between all pixel features.
  • Iterative Grouping: Using an iterative, threshold-based algorithm to group pixels.
  • Smallest First Strategy: Seeding superpixels from the pixel with the lowest degree (the most "unique" feature group) helps capture small objects or fine details that might otherwise be merged into the background.

Superpixel-Guided Seeded Region Growing (RG)

Initial "seeds" are generated using standard CAMs. During training, the segmentation network's predictions are refined: if a superpixel has a high average probability for a certain class, the entire superpixel is assigned that label. This allows sparse, blurry CAM seeds to "expand" into the sharp, semantically consistent boundaries defined by the DINO-based superpixels.

Experimental Results

The researchers evaluated their method on two industry-standard benchmarks: PASCAL VOC 2012 (20 classes) and MS-COCO 2014 (80 classes).

Comparison of conventional superpixels vs. the proposed semantic-aware superpixels across various input images.

The proposed method achieved competitive results against state-of-the-art (SOTA) models:

  • PASCAL VOC 2012: Achieved 69.5% mIOU on the validation set and 70.1% on the test set, outperforming several recent methods with more complex architectures.
  • MS-COCO 2014: Achieved 44.8% mIOU, setting a strong performance bar for methods using only image-level labels.
Detailed view of the bird segmentation, highlighting the sharp boundaries achieved by the proposed method.

Qualitative results demonstrate that the model produces much sharper object boundaries compared to standard CAM-based methods, successfully capturing thin structures like bird legs or bicycle spokes.

Key Insights and Discussion

  • Beyond Locality: Unlike traditional superpixels, these DINO-based groups can cluster semantically similar pixels even if they are not spatially connected, providing global awareness.
  • Synergy of Self-Supervision: The paper demonstrates that self-supervised models like DINO have already "learned" object shapes; WSSS simply needs to provide the class names. This removes the "hidden" requirement for pixel-level labels in saliency-based WSSS.

Limitations

Certain classes, such as "sofa" or "dining table," remain challenging because initial seeds often fail to expand correctly into the superpixels. Furthermore, the method's success is heavily tied to the quality of the DINO features; if the self-supervised model fails to distinguish between adjacent objects, the superpixel discovery will also fail.

Conclusion

By replacing external saliency detectors with semantic-aware superpixels derived from self-supervised features, this work provides a more "pure" approach to weakly supervised segmentation. Future work may explore these superpixels for unsupervised object detection or advanced saliency tasks.

Summary updated

BibTeX
@inproceedings{kim2023semantic,
  author = {Sangtae Kim and Daeyoung Park and Byonghyo Shim},
  title = {{Semantic-Aware Superpixel for Weakly Supervised Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25196},
}
Figure