Abstract
Weakly Supervised Semantic Segmentation (WSSS) aims to train segmentation models using only image-level tags, significantly reducing labeling costs. However, standard Class Activation Maps (CAMs) often suffer from discriminative bias and poor boundary adherence. This paper introduces S2C (From-SAM-to-CAMs), a framework that incorporates the Segment Anything Model (SAM) directly into the training loop. By utilizing SAM for feature-level guidance and high-quality self-supervision, S2C achieves a new state-of-the-art on major benchmarks, effectively bridging the gap between weak and full supervision.
Introduction
The primary challenge in WSSS is the "garbage in, garbage out" nature of initial seeds. Traditional CAMs tend to highlight only the most discriminative parts of an object and lack precise boundaries. While post-processing with SAM can improve results, it is limited by the quality of the initial CAM. S2C addresses this by transferring SAM’s boundary knowledge into the classifier during training, ensuring the model learns to produce high-quality CAMs natively.
Methodology
The S2C framework leverages SAM through two primary modules that exploit its capabilities at different levels of the training process:
SAM-segment Contrasting (SSC)
Operating at the feature level, SSC uses SAM in "segment-everything" mode to partition images into class-agnostic segments. Contrastive learning is then applied to ensure pixels within the same SAM segment have similar features, while pixels in different segments are pushed apart in the latent space. This forces the classifier to respect object boundaries from the earliest stages of feature extraction.
CAM-based Prompting Module (CPM)
At the logit level, the CPM identifies local maxima in CAMs to serve as point prompts for SAM. SAM then generates precise masks. To mitigate errors from noisy CAM seeds, a reliability metric—combining SAM's stability score and CAM activation—is used to aggregate these masks into a robust self-supervision signal that guides the model's training.
Experimental Results
The authors evaluated S2C on the PASCAL VOC 2012 and MS COCO 2014 datasets, two industry-standard benchmarks for object segmentation.
Performance Benchmarks
S2C achieved a significant performance leap, reaching 78.2% mIoU on the PASCAL VOC 2012 validation set, an improvement of approximately 7.4% over previous state-of-the-art methods like CLIP-ES. Qualitative analysis confirms that S2C is particularly effective at capturing complex structures and small objects that traditional WSSS methods often miss.
Discussion and Conclusion
S2C demonstrates that SAM is most effective when used as a "teacher" during training rather than a simple post-processing filter. By marrying the semantic knowledge of CAMs with the geometric precision of SAM, the framework produces superior pseudo-labels. While the computational cost of running SAM during training is a consideration, the gains in accuracy and robustness to complex scenes make S2C a powerful new approach for label-efficient semantic segmentation. Future work may involve integrating multi-modal models to further refine the prompting process.
Summary updated