Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation

Tao Chen, Yazhou Yao, Jinhui Tang

Abstract

Weakly Supervised Semantic Segmentation (WSSS) aims to reduce the heavy burden of pixel-level annotation by training models using only image-level labels. Traditional approaches rely on Class Activation Maps (CAMs), which often suffer from incomplete object coverage and require complex, multi-stage refinement. This paper presents the Multi-Granularity Denoising and Bidirectional Alignment (MDBA) framework, a single-stage approach that leverages off-the-shelf saliency maps. By introducing image-level and pixel-level noise filtering alongside a bidirectional alignment strategy, MDBA effectively bridges the gap between simple saliency-based labels and complex multi-object scenes. Experimental results on PASCAL VOC 2012 and MS COCO demonstrate that MDBA achieves state-of-the-art performance while maintaining high computational efficiency.

Introduction

Semantic segmentation is a fundamental computer vision task that requires assigning a category label to every pixel. However, the high cost of manual pixel-level annotation has led to increased interest in Weakly Supervised Semantic Segmentation (WSSS). Most existing WSSS methods utilize Class Activation Maps (CAMs) as initial seeds. While effective at identifying discriminative regions, CAMs typically fail to capture the full extent of an object, necessitating slow and complex multi-stage expansion processes.

An alternative is to use saliency maps, which provide better object boundaries but are inherently class-agnostic and noisy. Saliency maps excel in simple images with a single dominant object but struggle in complex scenes with multiple categories. The MDBA framework addresses these challenges through two core innovations: multi-granularity denoising to clean the saliency-based pseudo-labels and bidirectional alignment to generalize the model from simple to complex data distributions.

Methodology

The MDBA framework is designed to work in a single training stage, significantly reducing the overhead compared to multi-stage CAM-based methods. The architecture consists of two primary pillars: denoising and alignment.

Multi-Granularity Denoising

To handle the inherent noise in saliency maps, MDBA employs two distinct filtering modules:

  • Online Noise Filtering (ONF): Operating at the image level, ONF calculates a noise ratio for each image during training. If the pseudo-label deviates significantly from the network's prediction—based on a class-adaptive threshold—the image is excluded from that training iteration. This prevents the model from overfitting to "garbage" data.
  • Progressive Pixel-Level Noise Detection (PND): Operating at the pixel level, PND utilizes a dynamic loss threshold that becomes increasingly stringent as training progresses. Pixels exhibiting a cross-entropy loss above this threshold are masked out, allowing the network to prioritize learning from "clean" pixels before tackling more ambiguous regions.

Bidirectional Alignment

To overcome the class-agnostic nature of saliency maps and the distribution gap between simple and complex images, MDBA uses a bidirectional strategy:

  • Simple-to-Complex (S2C) Input Alignment: Using the CutMix strategy, the model physically pastes objects from simple, single-class images into new synthetic "complex" images. This provides the network with multi-object training samples where the ground truth is known, bridging the gap between simple saliency maps and complex real-world scenes.
  • Complex-to-Simple (C2S) Output Alignment: This module employs adversarial learning. A discriminator is trained to distinguish between predictions made on complex images and the cleaner pseudo-labels of simple images. This forces the segmentation network to produce high-quality, well-defined masks even for difficult, multi-class scenes.
Overview of the MDBA framework featuring Online Noise Filtering (ONF) and Bidirectional Alignment.

Key Results and Efficiency

MDBA was evaluated on the PASCAL VOC 2012 and MS COCO benchmarks. On PASCAL VOC 2012, the model achieved a mean Intersection over Union (mIoU) of 69.5% on the validation set and 70.2% on the test set. These results significantly outperform previous single-stage leaders like SSSS and RRM, often by margins exceeding 5% mIoU.

Beyond accuracy, MDBA is remarkably efficient. It utilizes only 47 million parameters, a fraction of the 124M and 138M parameters required by RRM and SSSS, respectively. Furthermore, while competing methods often require high-end hardware with up to 72GB of VRAM, MDBA operates comfortably on a standard 24GB GPU. When extended to a two-stage method—using MDBA to generate labels for a second round of training—the performance reaches 73.5% mIoU, surpassing many contemporary multi-stage CAM-based approaches.

Qualitative comparison of MDBA against other WSSS methods, showing superior object coverage and boundary adherence.

Conclusion

The MDBA framework demonstrates that saliency maps can serve as a highly effective shortcut for WSSS if properly denoised and aligned. By bypassing the traditional CAM-refinement pipeline, MDBA offers a more direct and computationally efficient path to high-quality semantic segmentation. Future work may involve integrating these denoising techniques with CAM-based methods to combine class-specific localization with the superior boundary awareness of saliency maps.

Summary updated

BibTeX
@article{chen2023multi,
  author = {Tao Chen and Yazhou Yao and Jinhui Tang},
  title = {{Multi-Granularity Denoising and Bidirectional Alignment for Weakly Supervised Semantic Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  doi = {10.1109/tip.2023.3275913},
}

Loading related content...

Figure