PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision

R. Austin McEver, B. S. ManjunatharXiv 2020 · OpenReview

TL;DR

This paper introduces a novel method for semantic segmentation using only point-level annotations, which are significantly cheaper to obtain than pixel-level labels. By integrating point supervision into the training of Class Activation Maps (CAMs) to create "Point-supervised CAMs" (PCAMs), the authors achieve state-of-the-art results for point-supervised segmentation, even outperforming methods that use stronger supervision like bounding boxes.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires extensive pixel-level annotations. This is a costly and time-consuming process, especially in specialized domains where expert knowledge is needed for accurate labeling. While weakly supervised methods using image-level labels have shown promise, they often struggle to achieve performance comparable to fully supervised approaches. The authors identify a gap in exploring "point-level" supervision – a middle ground where annotators mark a few points per object – which offers a more efficient annotation strategy without sacrificing too much localization information. They aim to leverage this cost-effective supervision to achieve high-performance semantic segmentation.

Methods

The proposed method, PCAMs, involves a multi-stage process:

  1. Point-supervised Class Activation Maps (PCAMs): The core innovation is integrating point annotations directly into the training of a CNN for classification. Instead of just using image-level labels, the network is trained to produce Class Activation Maps (CAMs) that are guided by the provided points. This is achieved by upsampling the CAMs during training and computing a mean squared error loss between the upsampled CAM and the ground truth point labels. This step significantly improves the localization accuracy of the activation maps compared to standard CAMs.
  2. IRNet for Affinity Mining and Refinement: The improved PCAMs are then used to mine "semantic affinity labels." These labels indicate whether pairs of pixels are likely to belong to the same class or different classes. This information is fed into a network called IRNet, which learns to predict a displacement vector field and a class boundary map. These outputs are used to refine the PCAMs, creating more precise "pseudo semantic segmentation labels."
  3. DeepLabv3+ for Final Segmentation: Finally, a standard, fully supervised semantic segmentation network (DeepLabv3+) is trained using these generated pseudo labels. This approach leverages the power of a strong segmentation architecture while being trained on the weakly supervised output. A key benefit is that during inference, only the trained DeepLabv3+ model needs to be used, simplifying the deployment process compared to methods requiring multiple models.

Datasets

The experiments were conducted on the PASCAL VOC 2012 dataset. For training, the PASCAL VOC 2012 training images were supplemented with images from the SBD dataset. The authors used point-level labels provided by a previous work [5] for training the PCAM network.

Key Insights and Ideas

  • Point Supervision is Underexplored but Powerful: The paper demonstrates that point supervision, a seemingly sparse form of annotation, can yield surprisingly strong results for semantic segmentation, rivaling and even surpassing methods with more complex supervision.
  • Integrating Points into CAM Training is Key: Directly incorporating point annotations into the CAM training process (as opposed to using them post-hoc) significantly enhances the localization capabilities of the activation maps.
  • Pseudo-Labeling for State-of-the-Art Architectures: Using refined PCAMs as pseudo-ground truth to train a powerful, fully supervised network like DeepLabv3+ is an effective strategy to achieve high performance without requiring full pixel-level annotations during the initial training phase.
  • Simplified Inference: By training a final, standard segmentation network on pseudo-labels, the inference pipeline becomes much simpler, requiring only one model, which is a practical advantage.

Key Results

  • The proposed PCAM method achieves state-of-the-art performance on the PASCAL VOC 2012 dataset for point-supervised semantic segmentation.
  • The final model, DeepLabv3+ trained on refined PCAMs, achieves a mean Intersection over Union (mIOU) of 70.5% on the PASCAL VOC 2012 validation set.
  • This performance outperforms state-of-the-art methods that use stronger supervision like bounding boxes and squiggles. For instance, it surpasses methods using bounding box supervision by a significant margin.
  • The method recovers an impressive 89% of the performance of its fully supervised counterpart using only point supervision.
  • Qualitative results show that PCAMs provide better localization and coverage of object extents compared to standard CAMs.

Limitations and Future Work

  • Choppy Edges: The paper notes that the label propagation via IRNet's random walk can sometimes result in "choppy edges" in the segmentation masks.
  • Missing Small Objects: While the PCAM-trained CNN makes fewer predictions, it can sometimes miss smaller objects, as observed in one of the qualitative examples.
  • Future Work: The authors suggest that their method and PCAMs can be integrated into other pipelines. They also imply that further refinement of the label propagation or segmentation stages could address the observed limitations. The paper also mentions that their code will be made available, which could facilitate further research in this area.

Summary updated

Abstract

Current state of the art methods for generating semantic segmentation rely heavily on a large set of images that have each pixel labeled with a class of interest label or background. Coming up with such labels, especially in domains that require an expert to do annotations, comes at a heavy cost in time and money. Several methods have shown that we can learn semantic segmentation from less expensive image-level labels, but the effectiveness of point level labels, a healthy compromise between all pixels labelled and none, still remains largely unexplored. This paper presents a novel procedure for producing semantic segmentation from images given some point level annotations. This method includes point annotations in the training of a convolutional neural network (CNN) for producing improved localization and class activation maps. Then, we use another CNN for predicting semantic affinities in order to propagate rough class labels and create pseudo semantic segmentation labels. Finally, we propose training a CNN that is normally fully supervised using our pseudo labels in place of ground truth labels, which further improves performance and simplifies the inference process by requiring just one CNN during inference rather than two. Our method achieves state of the art results for point supervised semantic segmentation on the PASCAL VOC 2012 dataset \cite{everingham2010pascal}, even outperforming state of the art methods for stronger bounding box and squiggle supervision.
BibTeX
@misc{mcever2020pcams,
  author = {R. Austin McEver and B. S. Manjunath},
  title = {{PCAMs: Weakly Supervised Semantic Segmentation Using Point Supervision}},
  year = {2020},
  eprint = {2007.05615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.05615},
}
Figure