Hierarchical Semantic Contrast for Weakly Supervised Semantic Segmentation

Yuanchen Wu, Xiaoqiang Li, Songmin Dai, Jide Li, Tong Liu, Shaorong Xie

Introduction

Weakly Supervised Semantic Segmentation (WSSS) aims to train segmentation models using only image-level labels, significantly reducing the cost of pixel-level annotation. The primary challenge in WSSS is the "discriminative region" problem associated with Class Activation Maps (CAMs). CAMs often only highlight the most salient parts of an object (e.g., a bird's head) while ignoring the rest of the body or including background noise. While contrastive learning has been applied to address this, most existing methods focus on a single semantic level, failing to capture the full structural context of a scene.

Methodology: Hierarchical Semantic Contrast (HSC)

This paper introduces Hierarchical Semantic Contrast (HSC), a framework that learns object patterns across three distinct semantic levels: Region of Interest (ROI), class, and pixel. By supervising these levels simultaneously, the model develops a more robust understanding of "objectness."

Visualizing the hierarchical semantic levels: ROI, class, and pixel-level contrast.

1. ROI-level Contrast

The ROI-level contrast distinguishes the general foreground from the background. This prevents the embedding space from collapsing and helps the model localize where objects generally exist within the scene.

2. Class-level Contrast

To ensure consistency across different images, the framework utilizes Momentum Prototypes. These prototypes act as a global memory, storing running averages of class features across the dataset. This ensures that features for a specific class (e.g., "cat") remain similar across different instances and distinct from other classes.

3. Pixel-level Contrast

Pixel-level contrast refines fine-grained details. To mitigate the noise inherent in weak labels, the authors employ a Conditional Random Field (CRF) to refine pseudo-labels before using them for contrastive supervision.

The architecture is built upon a Siamese Network where two augmented views of the same image are processed. Cross-view supervision is applied, where the hierarchical semantics from one view serve as the ground truth for the other, enforcing spatial and scale consistency.

Experimental Results

The HSC framework was evaluated on the PASCAL VOC 2012 benchmark. It achieved a new state-of-the-art performance with 73.6% mIoU on the validation set and 74.5% on the test set. Notably, when integrated into existing baselines like SEAM, HSC provided an 8.9% mIoU boost to the initial seed quality. Qualitative results demonstrate that HSC produces more complete masks with sharper boundaries, particularly in challenging categories like "plant" and "sofa."

Conclusion

Hierarchical Semantic Contrast offers a comprehensive approach to WSSS by leveraging multi-level semantic structures. By combining ROI, class, and pixel-level supervision within a siamese framework, HSC effectively addresses the limitations of standard CAM-based methods. Future work may involve integrating other forms of weak supervision, such as points or scribbles, to further narrow the gap with fully supervised methods.

Summary updated

Abstract

Weakly supervised semantic segmentation (WSSS) with image-level annotations has achieved great processes through class activation map (CAM). Since vanilla CAMs are hardly served as guidance to bridge the gap between full and weak supervision, recent studies explore semantic representations to make CAM fit for WSSS and demonstrate encouraging results. However, they generally exploit single-level semantics, which may hamper the model to learn a comprehensive semantic structure. Motivated by the prior that each image has multiple levels of semantics, we propose hierarchical semantic contrast (HSC) to ameliorate the above problem. It conducts semantic contrast from coarse-grained to fine-grained perspective, including ROI level, class level, and pixel level, making the model learn a better object pattern understanding. To further improve CAM quality, building upon HSC, we explore consistency regularization of cross supervision and develop momentum prototype learning to utilize abundant semantics across different images. Extensive studies manifest that our plug-and-play learning paradigm, HSC, can significantly boost CAM quality on both non-saliency-guided and saliency-guided baselines, and establish new state-of-the-art WSSS performance on PASCAL VOC 2012 dataset. Code is available at https://github.com/Wu0409/HSC_WSSS.
BibTeX
@inproceedings{wu2023hierarchical,
  author = {Yuanchen Wu and Xiaoqiang Li and Songmin Dai and Jide Li and Tong Liu and Shaorong Xie},
  title = {{Hierarchical Semantic Contrast for Weakly Supervised Semantic Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/171},
}
Figure