RecurSeed and CertainMix for Weakly Supervised Semantic Segmentation

Sang Hyun Jo, In-Jae Yu, Kyung-Su Kim

TL;DR

This paper introduces RecurSeed (RS) and EdgePredictMix (EPM), novel techniques to significantly improve weakly supervised semantic segmentation (WSSS) using only image-level labels. RS recursively refines object detection maps to reduce both missed detections and false positives, while EPM enhances data augmentation by focusing on object edges, leading to state-of-the-art performance on standard benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many computer vision applications. However, obtaining pixel-level annotations is extremely time-consuming and expensive. Weakly supervised semantic segmentation (WSSS) aims to alleviate this by using weaker forms of supervision, such as image-level labels. While promising, current WSSS methods struggle with two main issues: non-detection (failing to identify entire objects, especially large ones) and false detection (incorrectly labeling background regions as foreground, particularly for small objects). These issues, often termed "false positive/negative phenomena," limit performance and make existing methods complex to implement. Furthermore, standard data augmentation techniques can exacerbate these problems when applied to WSSS due to the inherent inaccuracies in the initial segmentation masks.

Methods

The paper proposes two core contributions:

  • RecurSeed (RS): This is a recursive learning process designed to iteratively refine class activation maps (CAMs). Unlike previous methods that treat CAM refinement as a post-processing step, RS integrates it into the training loop. It leverages both high-order feature correlations (like Self-Correlation map Generation - SCG) and image correlations (like Pixel-Adaptive Mask Refinement - PAMR) to simultaneously reduce false positives (FPs) and false negatives (FNs). The recursive nature allows the network to learn from its own refined outputs, progressively improving the segmentation masks.
  • EdgePredictMix (EPM): This is a novel data augmentation technique specifically designed for WSSS to address the "incomplete boundary information" issue. EPM focuses on refining object edges within predicted masks. It uses edge information derived from probability differences between adjacent pixels and then synthesizes new training samples by mixing images and their refined masks. This approach helps to better delineate object boundaries and improve the robustness of the segmentation.

The overall framework is a single-stage learning framework (SLF) that incorporates RS and EPM.

Datasets

The proposed methods were evaluated on two widely used datasets:

  • PASCAL VOC 2012: A standard benchmark for semantic segmentation.
  • MS COCO 2014: A larger and more challenging dataset with more categories.

The authors used augmented training sets for PASCAL VOC 2012 (10,582 images) and the standard training/validation splits for MS COCO 2014.

Key Insights and Ideas

  • Recursion for Refinement: The core idea of RecurSeed is that iteratively refining segmentation masks during training is more effective than treating it as a separate post-processing step. This allows the network to learn from its own improvements, leading to a more optimal solution.
  • Jointly Tackling FPs and FNs: The paper highlights that traditional methods often improve one type of error (FP or FN) at the expense of the other. RecurSeed's recursive approach is designed to address both simultaneously, which is crucial for robust WSSS.
  • Edge-Aware Data Augmentation: EdgePredictMix recognizes that standard data augmentation can be problematic with noisy WSSS masks. By focusing on edge refinement, EPM creates more reliable augmented data, leading to better performance, especially for object boundaries.
  • Single-Stage Simplicity, Multi-Stage Performance: The proposed single-stage framework (SLF) achieves performance comparable to or exceeding many multi-stage learning frameworks (MLFs), demonstrating that complexity isn't always necessary for state-of-the-art results.

Key Results

  • State-of-the-Art Performance: The proposed method (RSEPM) achieved new state-of-the-art results on both PASCAL VOC 2012 (74.4% mIoU on validation set) and MS COCO 2014 (46.4% mIoU on validation set).
  • Significant Improvements: RecurSeed alone provided a substantial boost in performance, reducing FPs and FNs. When combined with EdgePredictMix, the performance further increased.
  • Efficiency: The single-stage framework is efficient, outperforming previous methods with fewer parameters and without requiring extra complexity at inference time.
  • Qualitative Improvements: Visualizations show that RecurSeed progressively refines attention maps and pseudo masks, leading to more accurate object boundaries and complete object detection compared to baseline methods. EdgePredictMix further enhances these results.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper demonstrates that the method is not overly sensitive to Canny edge detection thresholds, the hyperparameters for the CertainFilter (dfg and dbg) do influence performance, and their optimal values might require tuning.
  • Generalizability to MLFs: The authors suggest that their proposed RS and EPM components can be integrated into existing multi-stage learning frameworks (MLFs) to potentially achieve even higher performance, indicating a direction for future research.
  • Exploring Different Backbones: While the proposed SLF achieves strong results with a standard ResNet-50 backbone, exploring more advanced backbones (like transformers) could further boost performance.

Summary updated

Abstract

Although weakly supervised semantic segmentation using only image-level labels (WSSS-IL) is potentially useful, its low performance and implementation complexity still limit its application. The main causes are (a) non-detection and (b) false-detection phenomena: (a) The class activation maps refined from existing WSSS-IL methods still only represent partial regions for large-scale objects, and (b) for small-scale objects, over-activation causes them to deviate from the object edges. We propose RecurSeed, which alternately reduces non- and false detections through recursive iterations, thereby implicitly finding an optimal junction that minimizes both errors. We also propose a novel data augmentation (DA) approach called EdgePredictMix, which further expresses an object's edge by utilizing the probability difference information between adjacent pixels in combining the segmentation results, thereby compensating for the shortcomings when applying the existing DA methods to WSSS. We achieved new state-of-the-art performances on both the PASCAL VOC 2012 and MS COCO 2014 benchmarks (VOC val: 74.4%, COCO val: 46.4%). The code is available at https://github.com/shjo-april/RecurSeed_and_EdgePredictMix.
BibTeX
@misc{jo2022recurseed,
  author = {Sang Hyun Jo and In-Jae Yu and Kyung-Su Kim},
  title = {{RecurSeed and CertainMix for Weakly Supervised Semantic Segmentation}},
  year = {2022},
  eprint = {2204.06754},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.06754},
}
Figure