From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation

Hyeokjun Kweon, Kuk-Jin Yoon

Abstract

Weakly Supervised Semantic Segmentation (WSSS) aims to train segmentation models using only image-level tags, significantly reducing labeling costs. However, standard Class Activation Maps (CAMs) often suffer from discriminative bias and poor boundary adherence. This paper introduces S2C (From-SAM-to-CAMs), a framework that incorporates the Segment Anything Model (SAM) directly into the training loop. By utilizing SAM for feature-level guidance and high-quality self-supervision, S2C achieves a new state-of-the-art on major benchmarks, effectively bridging the gap between weak and full supervision.

Introduction

The primary challenge in WSSS is the "garbage in, garbage out" nature of initial seeds. Traditional CAMs tend to highlight only the most discriminative parts of an object and lack precise boundaries. While post-processing with SAM can improve results, it is limited by the quality of the initial CAM. S2C addresses this by transferring SAM’s boundary knowledge into the classifier during training, ensuring the model learns to produce high-quality CAMs natively.

Methodology

The S2C framework leverages SAM through two primary modules that exploit its capabilities at different levels of the training process:

SAM-segment Contrasting (SSC)

Operating at the feature level, SSC uses SAM in "segment-everything" mode to partition images into class-agnostic segments. Contrastive learning is then applied to ensure pixels within the same SAM segment have similar features, while pixels in different segments are pushed apart in the latent space. This forces the classifier to respect object boundaries from the earliest stages of feature extraction.

The SAM-segment Contrasting (SSC) module uses SAM-defined segments to guide feature-level contrastive learning through a push-pull mechanism in the latent hypersphere.

CAM-based Prompting Module (CPM)

At the logit level, the CPM identifies local maxima in CAMs to serve as point prompts for SAM. SAM then generates precise masks. To mitigate errors from noisy CAM seeds, a reliability metric—combining SAM's stability score and CAM activation—is used to aggregate these masks into a robust self-supervision signal that guides the model's training.

Experimental Results

The authors evaluated S2C on the PASCAL VOC 2012 and MS COCO 2014 datasets, two industry-standard benchmarks for object segmentation.

Performance Benchmarks

S2C achieved a significant performance leap, reaching 78.2% mIoU on the PASCAL VOC 2012 validation set, an improvement of approximately 7.4% over previous state-of-the-art methods like CLIP-ES. Qualitative analysis confirms that S2C is particularly effective at capturing complex structures and small objects that traditional WSSS methods often miss.

Qualitative comparison showing how S2C's SAM-enhanced masks provide much closer alignment to Ground Truth (GT) than raw CAMs or standard SAM results.

Discussion and Conclusion

S2C demonstrates that SAM is most effective when used as a "teacher" during training rather than a simple post-processing filter. By marrying the semantic knowledge of CAMs with the geometric precision of SAM, the framework produces superior pseudo-labels. While the computational cost of running SAM during training is a consideration, the gains in accuracy and robustness to complex scenes make S2C a powerful new approach for label-efficient semantic segmentation. Future work may involve integrating multi-modal models to further refine the prompting process.

Summary updated

Abstract

Weakly Supervised Semantic Segmentation (WSSS) aims to learn the concept of segmentation using image-level class labels. Recent WSSS works have shown promising results by using the Segment Anything Model (SAM), a foundation model for segmentation, during the inference phase. However, we observe that these methods can still be vulnerable to the noise of class activation maps (CAMs) serving as initial seeds. As a remedy, this paper introduces FromSAM-to-CAMs (S2C), a novel WSSS framework that directly transfers the knowledge of SAM to the classifier during the training process, enhancing the quality of CAMs itself. S2C comprises SAM-segment Contrasting (SSC) and a CAM-based prompting module (CPM), which exploit SAM at the feature and logit levels, respectively. SSC performs prototype-based contrasting using SAM’s automatic segmentation results. It constrains each feature to be close to the prototype of its segment and distant from prototypes of the others. Meanwhile, CPM extracts prompts from the CAM of each class and uses them to generate classspecific segmentation masks through SAM. The masks are aggregated into unified self-supervision based on the conf idence score, designed to consider the reliability of both SAM and CAMs. S2C achieves a new state-of-the-art performance across all benchmarks, outperforming existing studies by significant margins.
BibTeX
@inproceedings{kweon2024from,
  author = {Hyeokjun Kweon and Kuk-Jin Yoon},
  title = {{From SAM to CAMs: Exploring Segment Anything Model for Weakly Supervised Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01844},
}
Figure