Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation

Tao Chen, Yazhou Yao, Jinhui Tang

Abstract

Weakly Supervised Semantic Segmentation (WSSS) aims to reduce the heavy burden of pixel-level annotation by training models using only image-level labels. Traditional approaches rely on Class Activation Maps (CAMs), which often suffer from incomplete object coverage and require complex, multi-stage refinement. This paper presents the Multi-Granularity Denoising and Bidirectional Alignment (MDBA) framework, a single-stage approach that leverages off-the-shelf saliency maps. By introducing image-level and pixel-level noise filtering alongside a bidirectional alignment strategy, MDBA effectively bridges the gap between simple saliency-based labels and complex multi-object scenes. Experimental results on PASCAL VOC 2012 and MS COCO demonstrate that MDBA achieves state-of-the-art performance while maintaining high computational efficiency.

Introduction

Semantic segmentation is a fundamental computer vision task that requires assigning a category label to every pixel. However, the high cost of manual pixel-level annotation has led to increased interest in Weakly Supervised Semantic Segmentation (WSSS). Most existing WSSS methods utilize Class Activation Maps (CAMs) as initial seeds. While effective at identifying discriminative regions, CAMs typically fail to capture the full extent of an object, necessitating slow and complex multi-stage expansion processes.

An alternative is to use saliency maps, which provide better object boundaries but are inherently class-agnostic and noisy. Saliency maps excel in simple images with a single dominant object but struggle in complex scenes with multiple categories. The MDBA framework addresses these challenges through two core innovations: multi-granularity denoising to clean the saliency-based pseudo-labels and bidirectional alignment to generalize the model from simple to complex data distributions.

Methodology

The MDBA framework is designed to work in a single training stage, significantly reducing the overhead compared to multi-stage CAM-based methods. The architecture consists of two primary pillars: denoising and alignment.

Multi-Granularity Denoising

To handle the inherent noise in saliency maps, MDBA employs two distinct filtering modules:

  • Online Noise Filtering (ONF): Operating at the image level, ONF calculates a noise ratio for each image during training. If the pseudo-label deviates significantly from the network's prediction—based on a class-adaptive threshold—the image is excluded from that training iteration. This prevents the model from overfitting to "garbage" data.
  • Progressive Pixel-Level Noise Detection (PND): Operating at the pixel level, PND utilizes a dynamic loss threshold that becomes increasingly stringent as training progresses. Pixels exhibiting a cross-entropy loss above this threshold are masked out, allowing the network to prioritize learning from "clean" pixels before tackling more ambiguous regions.

Bidirectional Alignment

To overcome the class-agnostic nature of saliency maps and the distribution gap between simple and complex images, MDBA uses a bidirectional strategy:

  • Simple-to-Complex (S2C) Input Alignment: Using the CutMix strategy, the model physically pastes objects from simple, single-class images into new synthetic "complex" images. This provides the network with multi-object training samples where the ground truth is known, bridging the gap between simple saliency maps and complex real-world scenes.
  • Complex-to-Simple (C2S) Output Alignment: This module employs adversarial learning. A discriminator is trained to distinguish between predictions made on complex images and the cleaner pseudo-labels of simple images. This forces the segmentation network to produce high-quality, well-defined masks even for difficult, multi-class scenes.
Overview of the MDBA framework featuring Online Noise Filtering (ONF) and Bidirectional Alignment.

Key Results and Efficiency

MDBA was evaluated on the PASCAL VOC 2012 and MS COCO benchmarks. On PASCAL VOC 2012, the model achieved a mean Intersection over Union (mIoU) of 69.5% on the validation set and 70.2% on the test set. These results significantly outperform previous single-stage leaders like SSSS and RRM, often by margins exceeding 5% mIoU.

Beyond accuracy, MDBA is remarkably efficient. It utilizes only 47 million parameters, a fraction of the 124M and 138M parameters required by RRM and SSSS, respectively. Furthermore, while competing methods often require high-end hardware with up to 72GB of VRAM, MDBA operates comfortably on a standard 24GB GPU. When extended to a two-stage method—using MDBA to generate labels for a second round of training—the performance reaches 73.5% mIoU, surpassing many contemporary multi-stage CAM-based approaches.

Qualitative comparison of MDBA against other WSSS methods, showing superior object coverage and boundary adherence.

Conclusion

The MDBA framework demonstrates that saliency maps can serve as a highly effective shortcut for WSSS if properly denoised and aligned. By bypassing the traditional CAM-refinement pipeline, MDBA offers a more direct and computationally efficient path to high-quality semantic segmentation. Future work may involve integrating these denoising techniques with CAM-based methods to combine class-specific localization with the superior boundary awareness of saliency maps.

Summary updated

Abstract

Weakly supervised semantic segmentation (WSSS) models relying on class activation maps (CAMs) have achieved desirable performance comparing to the non-CAMs-based counterparts. However, to guarantee WSSS task feasible, we need to generate pseudo labels by expanding the seeds from CAMs which is complex and time-consuming, thus hindering the design of efficient end-to-end (single-stage) WSSS approaches. To tackle the above dilemma, we resort to the off-the-shelf and readily accessible saliency maps for directly obtaining pseudo labels given the image-level class labels. Nevertheless, the salient regions may contain noisy labels and cannot seamlessly fit the target objects, and saliency maps can only be approximated as pseudo labels for simple images containing single-class objects. As such, the achieved segmentation model with these simple images cannot generalize well to the complex images containing multi-class objects. To this end, we propose an end-to-end multi-granularity denoising and bidirectional alignment (MDBA) model, to alleviate the noisy label and multi-class generalization issues. Specifically, we propose the online noise filtering and progressive noise detection modules to tackle image-level and pixel-level noise, respectively. Moreover, a bidirectional alignment mechanism is proposed to reduce the data distribution gap at both input and output space with simple-to-complex image synthesis and complex-to-simple adversarial learning. MDBA can reach the mIoU of 69.5% and 70.2% on validation and test sets for the PASCAL VOC 2012 dataset. The source codes and models have been made available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/NUST-Machine-Intelligence-Laboratory/MDBA</uri> .
BibTeX
@article{chen2023multi,
  author = {Tao Chen and Yazhou Yao and Jinhui Tang},
  title = {{Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  doi = {10.1109/tip.2023.3275913},
}
Figure