ECS-Net: Improving Weakly Supervised Semantic Segmentation by Using Connections Between Class Activation Maps

Kunyang Sun, Haoqing Shi, Zhengming Zhang, Yongming HuangICCV 2021 · OpenReview

Abstract

Image-level weakly supervised semantic segmentation is a challenging task. Because classification networks tend to capture only the most prominent object features and are insensitive to over-activation, standard Class Activation Maps (CAMs) are often too sparse and rough to guide segmentation training effectively. Inspired by the observation that erasing distinguishing features forces networks to identify new ones from non-discriminative regions, we propose a novel weakly supervised method that leverages the relationships between CAMs. In this work, we apply features learned from erased images as segmentation supervision, driving the network to learn more robust representations. Specifically, object regions identified by CAM are first erased from the images. To provide supervision for the remaining regions, the Erased CAM Supervision Net (ECS-Net) generates pixel-level labels by predicting segmentation results for these processed images. We also design a noise suppression rule to select reliable labels. Our experiments on the PASCAL VOC 2012 dataset demonstrate that ECS-Net achieves 67.6% mIoU on the test set and 66.6% mIoU on the validation set using only image-level labels, outperforming previous state-of-the-art methods.

Introduction

Semantic segmentation—the task of assigning a category label to every pixel in an image—is traditionally a "data-hungry" field requiring expensive, pixel-level annotations. To lower costs, researchers use Weakly Supervised Semantic Segmentation (WSSS), which relies on simple image-level labels (e.g., "there is a cat in this photo") rather than full masks.

The standard bridge between image-level labels and pixel-level masks is the Class Activation Map (CAM). However, CAMs have two major flaws:

  1. Sparsity: Classification networks are "lazy"; they only focus on the most discriminative parts of an object (like a bird's head) to identify it, ignoring the rest of the body.
  2. Noise: They often suffer from "over-activation," where the heatmap bleeds into the background, leading to rough and inaccurate boundaries.

ECS-Net (Erased CAM Supervision Net) addresses these issues by leveraging the relationship between different CAMs. By erasing the most discriminative parts of an image and using the resulting activations as pseudo-labels to supervise the original model, the network is forced to "see" the entire object rather than just its most prominent features.

Methodology: The ECS-Net Framework

The authors propose ECS-Net, which uses a self-supervised loop to expand and refine activation maps. The core idea is to use the network's own discovery of secondary features to teach itself what the full object looks like.

The ECS-Net architecture featuring the erasing strategy and dual-branch supervision loop.

The framework consists of several key components:

  1. The Erasing Strategy: The model first generates a standard CAM for an image. It then identifies the high-response regions (the most obvious parts) and "erases" them using Gaussian blur.
  2. Dual-Branch Supervision: The original image and the erased image are fed into a shared-weight backbone. Because the most obvious features are gone in the erased image, the network is forced to find new, less-obvious features to maintain its classification accuracy.
  3. Pseudo-Label Generation: The activations from the erased image are treated as "segmentation supervision" for the original image.
  4. Noise Suppression: To prevent the model from learning from unreliable data, the authors implement a rule to ignore pixels in the erased regions and apply a threshold to select only the most confident pixels as labels.
  5. Multi-Dilated Overlay Module: To better capture object boundaries, a module with multiple dilation rates is added to expand the receptive field, allowing the network to integrate local and global context more effectively.

Experiments and Results

The model was trained and evaluated on the PASCAL VOC 2012 benchmark, a standard dataset for semantic segmentation containing 21 classes. Training was augmented with the Semantic Boundary Dataset (SBD), totaling 10,582 images.

ECS-Net achieved state-of-the-art results for image-level WSSS at the time of publication, reaching 67.6% mIoU on the PASCAL VOC 2012 test set and 66.6% on the validation set using a ResNet-38 backbone. It outperformed previous leading methods like SEAM (64.5% val) and BES (65.7% val).

Qualitative comparison showing how ECS-Net produces denser and more accurate activation maps compared to sparse baseline CAMs.

Qualitative results show that ECS-Net produces much cleaner masks that follow object boundaries (like the legs of a chair or the wings of a bird) far better than baseline CAM methods. By supervising the original CAM with features learned from erased images, the network develops a more robust representation of the entire object.

Key Insights

  • Supervision over Merging: Unlike previous "adversarial erasing" methods that simply merged multiple CAMs together, ECS-Net uses the erased CAM to supervise the original one. This creates a stronger gradient for the network to learn robust representations.
  • Scale Sensitivity: The authors found that rescaling the first input image to be smaller ($\beta = 0.5$) improved performance. Smaller images provide a more global view, helping the model localize objects more accurately before the erasing process begins.
  • Reliability: The "Noise Suppression" step is vital. By acknowledging that erased regions are inherently "missing" features, the model avoids the trap of trying to learn from regions it has intentionally degraded.

Conclusion

ECS-Net demonstrates that the connections between Class Activation Maps can be a powerful source of supervision in weakly supervised settings. While the current approach is a two-stage process (pseudo-label generation followed by segmentation training), it significantly closes the gap between weakly supervised and fully supervised performance. Future work may explore more complex erasing strategies or end-to-end single-stage architectures.

Summary updated

Abstract

Image-level weakly supervised semantic segmentation is a challenging task. As classification networks tend to capture notable object features and are insensitive to over-activation, class activation map (CAM) is too sparse and rough to guide segmentation network training. Inspired by the fact that erasing distinguishing features force networks to collect new ones from non-discriminative object regions, we using relationships between CAMs to propose a novel weakly supervised method. In this work, we apply these features, learned from erased images, as segmentation super-vision, driving network to study robust representation. In specifically, object regions obtained by CAM techniques are erased on images firstly. To provide other regions with seg-mentation supervision, Erased CAM Supervision Net (ECS-Net) generates pixel-level labels by predicting segmentation results of those processed images. We also design the rule of suppressing noise to select reliable labels. Our experiments on PASCAL VOC 2012 dataset show that without data annotations except for ground truth image-level labels, our ECS-Net achieves 67.6% mIoU on test set and 66.6% mIoU on val set, outperforming previous state-of-the-art methods.
BibTeX
@inproceedings{sun2021ecs,
  author = {Kunyang Sun and Haoqing Shi and Zhengming Zhang and Yongming Huang},
  title = {{ECS-Net: Improving Weakly Supervised Semantic Segmentation by Using Connections Between Class Activation Maps}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=c-XfI_t4QO6},
}
Figure