Feature-Level and Spatial-Level Activation Expansion for Weakly-Supervised Semantic Segmentation

Junsu Choi, Jin-Seop Lee, Noo-ri Kim, SuHyun Yoon, Jee-Hyong LeeWACV 2025

TL;DR

This paper introduces FSAE, a novel framework for weakly-supervised semantic segmentation (WSSS) that tackles the issue of limited object coverage in traditional Class Activation Maps (CAMs). By expanding CAMs both at the feature level and spatially, FSAE generates more comprehensive and accurate pseudo-labels, leading to state-of-the-art segmentation performance without requiring expensive pixel-wise annotations.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires extensive pixel-level annotations. This is a costly and time-consuming process. Weakly-supervised semantic segmentation (WSSS) aims to alleviate this burden by training models with less precise labels, such as image-level tags, bounding boxes, or scribbles. Image-level WSSS is particularly attractive due to its minimal labeling cost.

A common approach in image-level WSSS involves a two-stage process: first, training a classification model to generate Class Activation Maps (CAMs), and second, using these CAMs as pseudo-labels to train a segmentation model. However, CAMs generated by classification models tend to focus only on the most discriminative parts of an object, leading to incomplete coverage and ultimately hindering segmentation performance. Existing methods have tried to address this by boosting feature learning or imposing consistency, but they often struggle to activate non-discriminative areas and accurately delineate object boundaries. This paper aims to bridge this gap by explicitly encouraging CAMs to cover the entire object, including its less discriminative regions and boundaries.

Methods

The proposed FSAE (Feature-level and Spatial-level Activation Expansion) framework consists of two main components:

  • Feature-level Activation Expansion (FAE): This component addresses the limited focus of standard CAMs by leveraging a weak-strong consistency strategy. It generates two views of an image: a weakly augmented view and a strongly augmented view. The pixel-wise predictions from the weakly augmented view are used as online pseudo-labels to supervise the strongly augmented view. This encourages the model to learn consistent predictions across diverse visual features within an object, thereby expanding the activation to more integral object areas. A confidence threshold is applied to filter out unreliable predictions, ensuring the quality of the generated pseudo-labels.
  • Spatial-level Activation Expansion (SAE): While FAE expands activation based on visual features, it may still miss ambiguous low-confidence regions, particularly around object boundaries. SAE addresses this by spatially expanding the pseudo-label regions using a dilation operation. This leverages the assumption that adjacent pixels share similar semantics. A confidence threshold is also applied to the dilated regions to prevent the inclusion of noisy predictions. The final expanded confidence mask is a union of the masks from FAE and SAE.

The FSAE framework then uses these expanded pseudo-labels to train the segmentation model via a cross-entropy loss. The method is designed to be easily incorporated into existing WSSS frameworks.

Datasets

The authors evaluate their method on two widely used benchmarks:

  • PASCAL VOC 2012: A dataset with 20 foreground classes and 1 background class.
  • MS COCO 2014: A larger dataset with 80 foreground classes and 1 background class.

They use the augmented training set for PASCAL VOC 2012 and the standard training/validation sets for MS COCO 2014.

Key Insights and Ideas

  • Weak-Strong Consistency for Feature Expansion: The core idea of using weak-strong augmentation to enforce consistent predictions across different visual representations of an object is a powerful way to encourage broader activation beyond the most discriminative parts.
  • Spatially Expanding Pseudo-Labels for Boundaries: Recognizing that low-confidence regions often lie at object boundaries, the paper proposes a simple yet effective dilation strategy to explicitly capture these crucial areas. This is a key insight for improving boundary segmentation.
  • Modular Design: FSAE is designed to be easily integrated with existing WSSS methods, allowing researchers to leverage its benefits without completely redesigning their pipelines. This makes it a practical contribution to the field.
  • Quality over Quantity for Pseudo-Labels: The use of confidence thresholds in both FAE and SAE highlights the importance of generating high-quality pseudo-labels, even if it means a smaller initial set, to avoid propagating noise and degrading performance.

Key Results

The FSAE framework consistently improves the performance of various baseline WSSS models (PPCw/EPS, SIPE, and MCTformer) on both PASCAL VOC 2012 and MS COCO 2014 datasets.

  • State-of-the-Art Performance: The paper reports achieving new state-of-the-art results on benchmark WSSS datasets. For instance, when combined with PPCw/EPS on PASCAL VOC 2012, FSAE achieved a significant performance gain, reaching 74.5% mIoU for initial seed quality and 77.0% for pseudo-mask quality.
  • Improved Seed Quality: FSAE significantly boosts the quality of initial seeds and pseudo-masks, as evidenced by substantial improvements in mIoU scores compared to baseline methods.
  • Reduced False Positives and Negatives: Table 5 shows that FSAE effectively reduces both False Positive Rate (FPR) and False Negative Rate (FNR) of pseudo-masks, indicating better alignment with actual object regions.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameters, it notes that the performance can be sensitive to the confidence thresholds (TFAE and TSAE). The optimal values might vary across datasets or models.
  • Computational Cost: While FSAE is designed to be efficient, the weak-strong consistency and dilation operations do add some computational overhead compared to simpler methods.
  • Future Work: The authors suggest that their method can be further improved by exploring more sophisticated expansion strategies or by integrating it with other advanced WSSS techniques. The modularity of FSAE also opens avenues for its application in other weakly supervised vision tasks.

Summary updated

BibTeX
@inproceedings{choi2025feature,
  author = {Junsu Choi and Jin-Seop Lee and Noo-ri Kim and SuHyun Yoon and Jee-Hyong Lee},
  title = {{Feature-Level and Spatial-Level Activation Expansion for Weakly-Supervised Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure