Learning Saliency-Free Model with Generic Features for Weakly-Supervised Semantic Segmentation

Wenfeng Luo, Meng YangAAAI 2020 · DOI

Introduction

Semantic segmentation is a fundamental task in computer vision that involves assigning a class label to every pixel in an image. While deep learning has achieved remarkable success in this area, the requirement for dense pixel-level annotations remains a significant bottleneck due to the high cost and time required for labeling. Weakly-supervised semantic segmentation (WSSS) addresses this by training models using only image-level labels, which are far easier to obtain.

The current standard for WSSS relies on Class Activation Mapping (CAM) to generate initial pseudo-labels. However, CAM suffers from two critical limitations: it typically only identifies the most discriminative parts of an object (e.g., the head of a bird rather than the whole body) and it lacks information about the background. To compensate, most existing methods rely on external saliency detectors. These detectors often require additional pixel-level data for training, which contradicts the goal of a truly "weakly" supervised system.

Discovering Class Patterns (DCP)

This paper proposes a self-contained, saliency-free approach that extracts high-quality proxy ground-truth masks directly from a classification network. The core innovation lies in Discovering Class Patterns (DCP) using generic features from intermediate layers of the network.

The "Goldilocks Zone" of Features

The authors observed a trade-off between feature depth and representation. While the final layers of a ResNet-101 are optimized for discriminative classification, they lose the integral shape of objects. Conversely, very early layers are too noisy. The researchers identified that intermediate "generic" features (specifically from the C41 block) provide the ideal balance, capturing the full extent and boundaries of objects.

Comparison of CAM vs. the proposed method, alongside PCA visualizations of feature distributions across different network blocks (C41, C42, C43).

PCA as a Co-localization Tool

To isolate the object from the background within these generic features, the method groups training images by class and applies Principal Component Analysis (PCA) to the feature maps. The first principal component represents the shared "class pattern" across the dataset. By projecting an image's features onto this discovered pattern, the model can estimate per-pixel masks:

  • Positive correlation: Indicates foreground pixels.
  • Negative/No correlation: Indicates background pixels.
  • Uncertainty: Pixels correlating with multiple classes are ignored during the initial training phase to prevent noise.

Experimental Results

The proposed method was evaluated on the PASCAL VOC 2012 and Microsoft COCO benchmarks. On PASCAL VOC 2012, the model achieved a mean Intersection over Union (mIoU) of 64.5% on the validation set and 64.6% on the test set. This performance significantly outperforms previous state-of-the-art methods like FickleNet (61.2%) and DSRG (59.0%), notably without the need for external saliency data.

Qualitative results demonstrate that while CAM only highlights small "hotspots," the DCP method produces masks that cover the entire body of animals and the full extent of vehicles, leading to much more accurate segmentation boundaries.

Conclusion and Future Work

By moving away from sparse CAMs and external saliency priors, this work demonstrates that classification networks already contain the necessary information for dense segmentation within their intermediate layers. Future research will focus on improving the detection of very small objects and developing more sophisticated methods to resolve multi-class conflicts in overlapping regions.

Summary updated

BibTeX
@inproceedings{luo2020learning,
  author = {Wenfeng Luo and Meng Yang},
  title = {{Learning Saliency-Free Model with Generic Features for Weakly-Supervised Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6842},
}
Figure