Representation Mutual Learning for End-to-End Weakly-Supervised Semantic Segmentation

Rongtao Xu, Changwei Wang, Shibiao Xu, Weiliang Meng, Xiaopeng ZhangOpenReview 2022 · OpenReview

TL;DR

This paper introduces Representation Mutual Learning (RML), a novel decoder-free framework for weakly supervised semantic segmentation (WSSS) that directly predicts segmentation masks. By leveraging mutual learning across instance, feature, and pixel levels, RML significantly boosts segmentation accuracy without the overhead of traditional decoders.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, typically requires extensive pixel-level annotations. Acquiring these labels is labor-intensive and costly. Weakly Supervised Semantic Segmentation (WSSS) aims to alleviate this by using cheaper, less precise labels like image-level tags. While recent end-to-end WSSS methods have shown promise, they often rely on segmentation branches or decoders, increasing model complexity and inference time. This paper tackles the challenge of achieving high WSSS performance with an efficient, end-to-end, decoder-free approach.

Methods

The core of this work is the Representation Mutual Learning (RML) framework. Unlike traditional methods that use decoders to refine initial predictions, RML directly learns segmentation masks by enabling different levels of feature representations to learn from each other. This is achieved through three key strategies:

  • CAM-driven Instance-leave Mutual Learning (CIML): This strategy focuses on improving the quality of Class Activation Maps (CAMs), which are initial heatmaps indicating object presence. CIML ensures that CAMs are "equivariant" (transform consistently with image transformations) and increases the separation between semantic prototypes of different object classes.
  • Multi-scale Feature-leave Mutual Learning (MFML): This component aims to enhance the understanding of contextual information within an image. It uses multi-scale features from different network layers, aligns their aggregated representations, and uses mutual information to preserve specific details while learning contextual relationships.
  • Affinity-aware Pixel-level Mutual Learning (APML): This strategy focuses on learning semantic affinity, which describes the relationship between adjacent pixels. It combines pixel-level mutual information loss with an affinity-aware contrastive learning loss to refine semantic affinity representations.

These three strategies work together, guided by a classification loss, to directly predict segmentation masks.

Datasets

The proposed RML framework is evaluated on two standard benchmarks for WSSS:

  • PASCAL VOC 2012: A dataset with 20 foreground object classes and 1 background class. The authors use an augmented version with the SBD dataset for training.
  • MS COCO 2014: A larger dataset comprising 80 foreground object classes and 1 background class.

The paper uses image-level labels as supervision for both datasets.

Key Insights and Ideas

  • Decoder-free is Viable: The paper demonstrates that it's possible to achieve state-of-the-art WSSS performance without relying on explicit segmentation branches or decoders, simplifying the model architecture and reducing computational cost.
  • Mutual Learning Across Levels is Powerful: The success of RML hinges on the synergistic interaction between instance-level (CAMs), feature-level (context), and pixel-level (affinity) representations. Each level contributes uniquely to improving the final segmentation.
  • CAM Quality is Crucial: Enhancing the initial CAMs through instance-level mutual learning directly translates to better segmentation masks, highlighting the importance of these initial cues.
  • Context and Affinity are Key Complementors: MFML and APML address the limitations of relying solely on local features by incorporating broader contextual understanding and fine-grained pixel relationships.

Key Results

On the PASCAL VOC 2012 validation set, RML achieves a mean Intersection over Union (mIoU) of 67.2%, significantly outperforming previous end-to-end methods. When combined with CRF post-processing, this further improves to 67.5%. On the MS COCO 2014 validation set, RML achieves 40.0% mIoU, also demonstrating strong performance against existing methods. The paper also highlights that RML is computationally more efficient than some prior methods, taking about 3 hours on a single NVIDIA 3090 GPU for the VOC dataset.

Limitations and Future Work

  • Post-processing: While RML achieves strong results, the performance is further boosted by CRF post-processing. Future work could aim to integrate such refinement capabilities directly into the end-to-end framework.
  • Hyperparameter Sensitivity: The paper mentions providing a sensitivity analysis for the loss weighting hyperparameters ($\alpha, \beta_1, \beta_2$) in the appendix, suggesting that tuning these might be important for optimal performance.
  • Generalization: While evaluated on standard benchmarks, exploring RML's performance on more diverse datasets or in different WSSS settings (e.g., scribbles, bounding boxes) could be a future direction.

The authors suggest that their RML framework provides a strong foundation for future research in efficient and effective WSSS.

Summary updated

Abstract

In recent years, end-to-end solutions for Weakly Supervised Semantic Segmentation (WSSS) with image-level labels have been developed rapidly. Previous end-to-end methods usually rely on segmentation branches or decoders to predict segmentation masks, bringing additional parameter numbers and consumption time. In this paper, we propose a decoder-free Representation Mutual Learning (RML) framework to directly predict segmentation masks, which leverages collaborative learning and mutual teaching among multi-level feature representations to improve segmentation performance. Our RML is a straightforward and efficient end-to-end WSSS framework, which incorporates the instance-level, feature-level and pixel-level representation mutual learning strategies to improve segmentation quality. To enhance the Class Activation Map (CAM) representations, we propose a CAM-driven Instance-leave Mutual Learning strategy that preserves the equivariance of CAMs and expands the distance between different classes of semantic prototypes. Besides, we design a Multi-scale Feature-leave Mutual Learning strategy, which can align aggregated contextual representations and facilitate the representation capability of contextual representations. Furthermore, we also provide an Affinity-aware Pixel-level Mutual Learning strategy to learn semantic affinity representations. Experiments validate that our RML yields a significant performance improvement over recent end-to-end methods on the Pascal VOC 2012 dataset and the MS COCO 2014 dataset. The release code is available at supplementary material.
BibTeX
@misc{xu2022representation,
  author = {Rongtao Xu and Changwei Wang and Shibiao Xu and Weiliang Meng and Xiaopeng Zhang},
  title = {{Representation Mutual Learning for End-to-End Weakly-Supervised Semantic Segmentation}},
  year = {2022},
  url = {https://openreview.net/forum?id=BnznzofWMi},
  howpublished = {OpenReview},
}
Figure