Abstract
Weakly Supervised Semantic Segmentation (WSSS) aims to train segmentation models using only image-level labels, significantly reducing annotation costs. However, standard Class Activation Maps (CAMs) often suffer from incompleteness and imprecision. This paper proposes the Adversarial Learning of Classifier and Reconstructor (ACR) framework. ACR frames precise segmentation as a state of mutual "un-inferability" between image regions. By training a classifier to generate segments and a reconstructor to attempt cross-segment reconstruction, the model learns to produce high-quality masks that cover entire objects and respect boundaries. ACR achieves state-of-the-art results on PASCAL VOC 2012 and MS COCO 2014 benchmarks.
Introduction
Semantic segmentation is a fundamental computer vision task that requires pixel-level labels for training. Obtaining such labels is labor-intensive and expensive. WSSS offers a more scalable alternative by utilizing image-level tags. The predominant approach in WSSS relies on CAMs to generate pseudo-labels. Despite their utility, CAMs typically highlight only the most discriminative object parts (e.g., a bird's head) and often "bleed" into the background because there is no pixel-level supervision to define object boundaries.
The ACR Framework
The core innovation of ACR is the concept of inferability. The authors posit that if an image is correctly segmented into a target object and its background, the visual information in one segment should not be sufficient to reconstruct the other. Conversely, if segmentation is imprecise, "remnants" of the object in the background segment (or vice versa) will allow a reconstructor to infer the missing parts.
Adversarial Game
ACR implements this intuition through an adversarial game between two networks:
- The Classifier ($F$): Generates CAMs to decompose the image into target and non-target segments.
- The Reconstructor ($G$): Attempts to reconstruct one segment using features from the other.
The training alternates between two phases:
- Reconstructor-Updating (RU) Phase: The reconstructor is trained to minimize reconstruction loss by identifying and exploiting any "remnants" left behind by the classifier.
- Classifier-Updating (CU) Phase: The classifier is updated to maximize the reconstruction loss. This forces the classifier to refine its CAMs, "hiding" remnants by ensuring the segments are as distinct as possible. This process encourages the CAM to expand to cover the full object and align with its true boundaries.
Stochastic Remnant Feeding (SRF)
To prevent the reconstructor from simply memorizing training images, the authors introduce Stochastic Remnant Feeding (SRF). This technique injects synthetic remnants (random patches) during training, forcing the reconstructor to learn the general mechanism of exploiting leaked information rather than relying on image-specific memory.
Experimental Evaluation
The authors evaluated ACR on the PASCAL VOC 2012 and MS COCO 2014 datasets, which are gold-standard benchmarks for semantic segmentation.
Key Results
- PASCAL VOC 2012: ACR achieved an mIoU of 71.9% on both validation and test sets, setting a new state-of-the-art for WSSS and outperforming previous leaders like MCT and AEFT.
- MS COCO 2014: The method reached an mIoU of 45.3%, demonstrating strong performance on complex scenes with many object categories.
- Ablation Study: The effectiveness of the adversarial component is clear: adding the reconstructor and SRF improved the baseline CAM mIoU from 48.4% to 60.3% before any post-processing—a massive improvement validating the core theory.
- Versatility: ACR is backbone-agnostic, showing consistent improvements when used with both ResNet (CNN) and Vision Transformer (ViT) architectures.
Conclusion
ACR presents a novel perspective on WSSS by treating segmentation as an information de-correlation task. By leveraging an adversarial reconstructor, the framework provides a continuous, self-supervised signal that effectively addresses the incompleteness and imprecision of traditional CAMs. While the training process is more complex than standard classification, the significant gains in segmentation quality and the lack of reliance on external saliency maps make ACR a powerful tool for weakly supervised learning.
Summary updated