TL;DR
This paper introduces RecurSeed (RS) and EdgePredictMix (EPM), novel techniques to significantly improve weakly supervised semantic segmentation (WSSS) using only image-level labels. RS recursively refines object detection maps to reduce both missed detections and false positives, while EPM enhances data augmentation by focusing on object edges, leading to state-of-the-art performance on standard benchmarks.
What Problem Does This Paper Address?
Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many computer vision applications. However, obtaining pixel-level annotations is extremely time-consuming and expensive. Weakly supervised semantic segmentation (WSSS) aims to alleviate this by using weaker forms of supervision, such as image-level labels. While promising, current WSSS methods struggle with two main issues: non-detection (failing to identify entire objects, especially large ones) and false detection (incorrectly labeling background regions as foreground, particularly for small objects). These issues, often termed "false positive/negative phenomena," limit performance and make existing methods complex to implement. Furthermore, standard data augmentation techniques can exacerbate these problems when applied to WSSS due to the inherent inaccuracies in the initial segmentation masks.
Methods
The paper proposes two core contributions:
- RecurSeed (RS): This is a recursive learning process designed to iteratively refine class activation maps (CAMs). Unlike previous methods that treat CAM refinement as a post-processing step, RS integrates it into the training loop. It leverages both high-order feature correlations (like Self-Correlation map Generation - SCG) and image correlations (like Pixel-Adaptive Mask Refinement - PAMR) to simultaneously reduce false positives (FPs) and false negatives (FNs). The recursive nature allows the network to learn from its own refined outputs, progressively improving the segmentation masks.
- EdgePredictMix (EPM): This is a novel data augmentation technique specifically designed for WSSS to address the "incomplete boundary information" issue. EPM focuses on refining object edges within predicted masks. It uses edge information derived from probability differences between adjacent pixels and then synthesizes new training samples by mixing images and their refined masks. This approach helps to better delineate object boundaries and improve the robustness of the segmentation.
The overall framework is a single-stage learning framework (SLF) that incorporates RS and EPM.
Datasets
The proposed methods were evaluated on two widely used datasets:
- PASCAL VOC 2012: A standard benchmark for semantic segmentation.
- MS COCO 2014: A larger and more challenging dataset with more categories.
The authors used augmented training sets for PASCAL VOC 2012 (10,582 images) and the standard training/validation splits for MS COCO 2014.
Key Insights and Ideas
- Recursion for Refinement: The core idea of RecurSeed is that iteratively refining segmentation masks during training is more effective than treating it as a separate post-processing step. This allows the network to learn from its own improvements, leading to a more optimal solution.
- Jointly Tackling FPs and FNs: The paper highlights that traditional methods often improve one type of error (FP or FN) at the expense of the other. RecurSeed's recursive approach is designed to address both simultaneously, which is crucial for robust WSSS.
- Edge-Aware Data Augmentation: EdgePredictMix recognizes that standard data augmentation can be problematic with noisy WSSS masks. By focusing on edge refinement, EPM creates more reliable augmented data, leading to better performance, especially for object boundaries.
- Single-Stage Simplicity, Multi-Stage Performance: The proposed single-stage framework (SLF) achieves performance comparable to or exceeding many multi-stage learning frameworks (MLFs), demonstrating that complexity isn't always necessary for state-of-the-art results.
Key Results
- State-of-the-Art Performance: The proposed method (RSEPM) achieved new state-of-the-art results on both PASCAL VOC 2012 (74.4% mIoU on validation set) and MS COCO 2014 (46.4% mIoU on validation set).
- Significant Improvements: RecurSeed alone provided a substantial boost in performance, reducing FPs and FNs. When combined with EdgePredictMix, the performance further increased.
- Efficiency: The single-stage framework is efficient, outperforming previous methods with fewer parameters and without requiring extra complexity at inference time.
- Qualitative Improvements: Visualizations show that RecurSeed progressively refines attention maps and pseudo masks, leading to more accurate object boundaries and complete object detection compared to baseline methods. EdgePredictMix further enhances these results.
Limitations and Future Work
- Hyperparameter Sensitivity: While the paper demonstrates that the method is not overly sensitive to Canny edge detection thresholds, the hyperparameters for the CertainFilter (dfg and dbg) do influence performance, and their optimal values might require tuning.
- Generalizability to MLFs: The authors suggest that their proposed RS and EPM components can be integrated into existing multi-stage learning frameworks (MLFs) to potentially achieve even higher performance, indicating a direction for future research.
- Exploring Different Backbones: While the proposed SLF achieves strong results with a standard ResNet-50 backbone, exploring more advanced backbones (like transformers) could further boost performance.
Summary updated