Abstract
This paper introduces a novel approach to Weakly Supervised Semantic Segmentation (WSSS) that eliminates the need for pixel-level saliency detectors. By leveraging features from a self-supervised Vision Transformer (DINO), the authors generate "semantic-aware superpixels" that capture object boundaries and global context. These superpixels guide a seeded region-growing process to train a high-performing segmentation network using only image-level labels.
Introduction
Semantic segmentation—assigning a class label to every pixel in an image—is a cornerstone of computer vision but typically requires massive datasets with dense, pixel-level annotations. Creating these is incredibly laborious. Weakly Supervised Semantic Segmentation (WSSS) attempts to train models using only "weak" labels, such as image-level tags (e.g., "this image contains a cat").
The standard tool for WSSS is the Class Activation Map (CAM), which identifies which pixels a classification network focuses on. However, CAMs are notoriously "blotchy," highlighting only the most discriminative parts of an object while missing boundaries and the full extent of the body. To address this, most state-of-the-art methods rely on external saliency detectors. However, these detectors often require pixel-level labels for their own training, which partially defeats the purpose of WSSS. This paper proposes a method to achieve high-quality boundaries using purely unsupervised features.
Methodology
The authors propose a two-stage framework that leverages self-supervised features to guide segmentation.
Semantic-Aware Superpixel Discovery
Instead of traditional color-based superpixels (like SLIC), the method uses features from DINO, a Vision Transformer trained via self-distillation without labels. DINO features are known to contain explicit information about object shapes. The process involves:
- Similarity Matrix: Calculating a similarity matrix between all pixel features.
- Iterative Grouping: Using an iterative, threshold-based algorithm to group pixels.
- Smallest First Strategy: Seeding superpixels from the pixel with the lowest degree (the most "unique" feature group) helps capture small objects or fine details that might otherwise be merged into the background.
Superpixel-Guided Seeded Region Growing (RG)
Initial "seeds" are generated using standard CAMs. During training, the segmentation network's predictions are refined: if a superpixel has a high average probability for a certain class, the entire superpixel is assigned that label. This allows sparse, blurry CAM seeds to "expand" into the sharp, semantically consistent boundaries defined by the DINO-based superpixels.
Experimental Results
The researchers evaluated their method on two industry-standard benchmarks: PASCAL VOC 2012 (20 classes) and MS-COCO 2014 (80 classes).
The proposed method achieved competitive results against state-of-the-art (SOTA) models:
- PASCAL VOC 2012: Achieved 69.5% mIOU on the validation set and 70.1% on the test set, outperforming several recent methods with more complex architectures.
- MS-COCO 2014: Achieved 44.8% mIOU, setting a strong performance bar for methods using only image-level labels.
Qualitative results demonstrate that the model produces much sharper object boundaries compared to standard CAM-based methods, successfully capturing thin structures like bird legs or bicycle spokes.
Key Insights and Discussion
- Beyond Locality: Unlike traditional superpixels, these DINO-based groups can cluster semantically similar pixels even if they are not spatially connected, providing global awareness.
- Synergy of Self-Supervision: The paper demonstrates that self-supervised models like DINO have already "learned" object shapes; WSSS simply needs to provide the class names. This removes the "hidden" requirement for pixel-level labels in saliency-based WSSS.
Limitations
Certain classes, such as "sofa" or "dining table," remain challenging because initial seeds often fail to expand correctly into the superpixels. Furthermore, the method's success is heavily tied to the quality of the DINO features; if the self-supervised model fails to distinguish between adjacent objects, the superpixel discovery will also fail.
Conclusion
By replacing external saliency detectors with semantic-aware superpixels derived from self-supervised features, this work provides a more "pure" approach to weakly supervised segmentation. Future work may explore these superpixels for unsupervised object detection or advanced saliency tasks.
Summary updated