Abstract
Weakly supervised semantic segmentation using only image-level labels (WSSS-IL) offers a cost-effective alternative to pixel-perfect annotation but often suffers from low performance due to non-detection of large objects and false-detection of small objects. We introduce two novel techniques: RecurSeed (RS), which recursively refines pseudo-labels to minimize both error types, and EdgePredictMix (EPM), an edge-aware data augmentation strategy that utilizes probability differences between adjacent pixels to improve boundary accuracy. Our approach achieves state-of-the-art results on PASCAL VOC 2012 (74.4% mIoU) and MS COCO 2014 (46.4% mIoU) benchmarks, demonstrating high efficiency in both single- and multi-stage frameworks.
Introduction
Semantic segmentation is a fundamental computer vision task that assigns class labels to every pixel. While fully supervised methods achieve high accuracy, they require labor-intensive pixel-level annotations. Weakly Supervised Semantic Segmentation (WSSS) addresses this by training models on cheaper "weak" labels, such as image-level tags.
The standard WSSS pipeline relies on Class Activation Maps (CAMs) to generate initial pseudo-labels. However, CAMs typically face two challenges:
- Non-detection: Models focus only on the most discriminative parts of an object (e.g., the head of a bird), leaving the rest of the object unsegmented.
- False-detection: For smaller objects, activations often "bleed" into the background, leading to inaccurate, over-expanded boundaries.
Existing solutions often involve complex, multi-stage pipelines that are difficult to optimize. Furthermore, standard data augmentation (DA) techniques like CutMix or ClassMix often fail in WSSS because the masks used for blending are themselves noisy and inaccurate at the edges.
Proposed Methodology
RecurSeed (RS)
RecurSeed integrates mask refinement directly into the training loop as a recursive process rather than a static post-processing step. It employs two complementary mechanisms:
- Self-Correlation map Generation (SCG): This leverages high-order (second-order) correlations between pixels to expand the activation regions for large objects, mitigating non-detection.
- Pixel-Adaptive Mask Refinement (PAMR): This uses local pixel affinities to sharpen object boundaries, reducing false-detection.
By iterating these processes, the network implicitly finds an optimal junction that balances expansion and sharpening, effectively bridging the gap between noisy CAMs and true object masks.
EdgePredictMix (EPM)
To further enhance boundary precision, we propose EdgePredictMix, a specialized data augmentation strategy. Unlike standard mixing methods that rely on potentially inaccurate masks, EPM focuses on the "uncertainty" at object edges.
- Uncertainty Identification: It identifies pixels at boundaries where the model's prediction is less confident.
- Superpixel-based Voting: Using Canny edge detection and Connected Component Labeling (CCL), it groups pixels and uses a voting mechanism to re-assign uncertain pixels to the most probable class.
- Refined Blending: Images are blended using these refined masks, providing the network with a much cleaner and more edge-accurate training signal.
Experimental Results
The proposed framework was evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets.
Performance and Efficiency
Our method achieved new state-of-the-art (SOTA) performance:
- PASCAL VOC 2012: 74.4% mIoU on the validation set.
- MS COCO 2014: 46.4% mIoU on the validation set.
Notably, our single-stage framework (SLF) outperformed many complex multi-stage models. For example, our ResNet-50 based model achieved competitive results with only 40M parameters, significantly more efficient than competing methods requiring over 124M parameters.
Qualitative Analysis
Visual results demonstrate that our method produces masks that closely align with ground truth, effectively capturing full object extents while maintaining sharp boundaries even in complex multi-object scenes.
Conclusion
RecurSeed and EdgePredictMix significantly advance the state-of-the-art in WSSS-IL by addressing the core issues of non-detection and false-detection through recursive refinement and edge-aware augmentation. The proposed single-stage framework offers a powerful balance of performance and implementation simplicity, making high-quality semantic segmentation more accessible with minimal annotation effort. Future work will explore extending these recursive principles to other forms of weak supervision, such as points or scribbles.
Summary updated