TL;DR
This paper introduces a novel coarse-to-fine framework that leverages pre-trained foundation models like CLIP and SAM to generate high-quality segmentation seeds for weakly supervised semantic segmentation (WSSS). By learning task-specific prompts and employing a SAM-based seeding module, the method significantly improves seed generation, leading to state-of-the-art performance on benchmark datasets.
What Problem Does This Paper Address?
Weakly supervised semantic segmentation (WSSS) aims to segment images using only image-level labels, which is significantly cheaper than pixel-level annotations required for fully supervised methods. However, a major challenge in WSSS is generating accurate "seeds" – initial segmentation masks that serve as pseudo-labels for training a segmentation network. Existing methods often rely on Class Activation Maps (CAMs), which tend to be imprecise, focusing on discriminative parts rather than complete objects and sometimes including background noise. While recent work has explored using powerful foundation models like CLIP and SAM, their integration into WSSS is still in its early stages, with opportunities for more effective prompt learning and seed refinement.
Methods
The core of this work is a coarse-to-fine framework that utilizes both CLIP and SAM. The framework involves two main stages:
- Prompt Learning: Instead of using manually designed prompts for CLIP, the authors propose learning task-specific prompts for two distinct tasks:
- Classification-specific prompts: These are learned using a multi-label contrastive loss, supervised by image-level labels. This helps CLIP identify relevant image regions for classification.
-
Segmentation-specific prompts: These are learned using a CAM activation loss, guided by the coarse seeds generated from the classification task. This refines the segmentation capabilities. The CLIP model itself remains frozen, making the learning process efficient.
-
SAM-based Seeding Module: This module takes the generated CAMs (refined by Softmax-GradCAM and class-aware attention-based affinity) and uses masks from the Segment Anything Model (SAM) to produce high-quality segmentation seeds. The module performs:
- Quasi-superpixel generation: It filters SAM's masks to prioritize "whole" object masks and minimize overlap, creating more robust superpixels.
- Quasi-superpixel classification: It assigns semantic labels to these superpixels using the CAMs.
- Seed map generation: It constructs the final seed map from the classified superpixels.
These generated seeds then serve as pseudo-labels to train an off-the-shelf segmentation network.
Datasets
The proposed method is evaluated on two standard datasets:
- PASCAL VOC 2012: Contains 20 object classes and 1 background class. An augmented training set of 10,582 images is used.
- MS COCO 2014: Contains 80 object classes and 1 background class.
Key Insights and Ideas
- Task-specific prompt learning is crucial: Simply using CLIP with manual prompts doesn't yield significant gains for WSSS. Learning distinct prompts for classification and segmentation tasks allows CLIP to adapt more effectively to the specific needs of WSSS.
- Coarse-to-fine refinement is effective: Using the output of a classification task to guide a segmentation task creates a powerful iterative refinement process for generating better seeds.
- SAM as a seed refinement tool: Leveraging SAM's powerful mask generation capabilities, especially with a focus on "whole" object masks, significantly improves the quality and completeness of segmentation seeds compared to traditional methods like dense CRF.
- Efficiency through frozen foundation models: Keeping CLIP's weights frozen and only learning prompts makes the training process efficient, allowing for faster adaptation to WSSS.
Key Results
- The proposed method achieves state-of-the-art performance on PASCAL VOC 2012, reaching an mIoU of 74.2% with the R101 backbone and 82.6% with the Swin-L backbone.
- It also achieves competitive results on MS COCO 2014, with an mIoU of 48.6% (R101) and 55.4% (Swin-L).
- Ablation studies demonstrate that the segmentation task significantly boosts performance over just the classification task.
- The SAM-based seeding module outperforms dense CRF for post-processing refinement.
- The specific prompt design (class-specific context, prepending foreground labels, and excluding background labels) is shown to be optimal.
Limitations and Future Work
- Reliance on pre-trained models: The performance is inherently tied to the quality and capabilities of the pre-trained CLIP and SAM models.
- Prompt learning efficiency: While efficient due to frozen weights, the learning of prompts still requires dedicated training.
- Future work: The authors suggest that their SAM-based seeding module can be further applied to refine final segmentation results, which they demonstrate in their experiments. The framework's adaptability to other foundation models and WSSS scenarios could also be explored.
Summary updated