Improve CAM with Auto-adapted Segmentation and Co-supervised Augmentation

Ziyi Kou, Guofeng Cui, Shaojie Wang, Wentian Zhao, Chenliang XuWACV 2021 · OpenReview

Abstract

Weakly Supervised Object Localization (WSOL) methods generate both classification and localization results by learning from only image category labels. Previous methods usually utilize class activation map (CAM) to obtain target object regions. However, most of them only focus on improving foreground object parts in CAM, but ignore the important effect of its background contents. In this paper, we propose a confidence segmentation (ConfSeg) module that builds confidence score for each pixel in CAM without introducing additional hyper-parameters. The generated sample-specific confidence mask is able to indicate the extent of determination for each pixel in CAM, and further supervises additional CAM extended from internal feature maps. Besides, we introduce Co-supervised Augmentation (CoAug) module to capture feature-level representation for foreground and background parts in CAM separately. Then a metric loss is applied at batch sample level to augment distinguish ability of our model, which helps a lot to localize more related object parts. Our final model, CSoA, combines the two modules and achieves superior performance, e.g. 37.69% and 48.81% Top-1 localization error on CUB-200 and ILSVRC datasets, respectively, which outperforms all previous methods and becomes the new state-of-the-art.

Introduction

The standard tool for Weakly Supervised Object Localization is the Class Activation Map (CAM), which highlights regions a neural network uses for classification. However, CAMs often suffer from a "laziness" problem: the network identifies only the most discriminative parts of an object (e.g., a bird's beak) rather than its full extent. While previous methods attempted to solve this by "erasing" discriminative parts, they often compromised classification accuracy and ignored the background context. This research introduces CSoA (Confidence Segmentation and Co-supervised Augmentation), a framework that leverages background information as a constraint to better define foreground boundaries.

Methodology

The CSoA framework integrates two primary modules into standard CNN backbones to address the limitations of traditional CAMs:

Confidence Segmentation (ConfSeg) Module

Unlike traditional methods that use fixed thresholds, ConfSeg generates a sample-specific confidence mask. By utilizing two CAMs from different network layers ($S_L$ and $S_F$), it identifies "ambiguous" pixels (near the mean activation) and "confident" pixels (far from the mean). The confident pixels from one map supervise the refinement of the other, allowing the model to explore object-related regions without drifting into the background.

Visualization of the CSoA pipeline showing raw input, intermediate masks ($S_L$, $S_F$), combined confidence mask, and final bounding box localization.

Co-supervised Augmentation (CoAug) Module

This module frames localization as a metric learning problem. It extracts feature vectors for predicted foreground and background regions and applies a loss function based on four key rules: foreground and background within an image must be distinct; foregrounds of the same category across images should be similar; foregrounds of different categories should be distinct; and backgrounds should not be misclassified as foreground.

Experimental Results

The CSoA model was evaluated on the CUB-200-2011 and ILSVRC (ImageNet) datasets, achieving significant improvements over existing state-of-the-art methods.

  • CUB-200-2011: The model achieved a 37.69% Top-1 localization error, a substantial improvement over the VGG baseline (53.42%) and surpassing previous leaders like DANet (47.48%).
  • ILSVRC: CSoA reached a 48.81% Top-1 localization error, outperforming methods such as ACoL and SPG.

Notably, CSoA maintains or improves classification integrity, avoiding the accuracy trade-offs common in "erasing-based" localization methods.

Conclusion

CSoA demonstrates that background context and dynamic confidence masks are powerful tools for refining object localization in weakly supervised settings. By using dual-classifier interaction and metric learning, the model captures more complete object extents. Future work will focus on making the final inference thresholding fully learnable and optimizing the framework for complex residual architectures like ResNet.

Summary updated

Abstract

Weakly Supervised Object Localization (WSOL) methods generate both classification and localization results by learning from only image category labels. Previous methods usually utilize class activation map (CAM) to obtain target object regions. However, most of them only focus on improving foreground object parts in CAM, but ignore the important effect of its background contents. In this paper, we propose a confidence segmentation (ConfSeg) module that builds confidence score for each pixel in CAM without introducing additional hyper-parameters. The generated sample-specific confidence mask is able to indicate the extent of determination for each pixel in CAM, and further supervises additional CAM extended from internal feature maps. Besides, we introduce Co-supervised Augmentation (CoAug) module to capture feature-level representation for foreground and background parts in CAM separately. Then a metric loss is applied at batch sample level to augment distinguish ability of our model, which helps a lot to localize more related object parts. Our final model, CSoA, combines the two modules and achieves superior performance, e.g. 37.69% and 48.81% Top-1 localization error on CUB-200 and ILSVRC datasets, respectively, which outperforms all previous methods and becomes the new state-of-the-art.
BibTeX
@inproceedings{kou2021improve,
  author = {Ziyi Kou and Guofeng Cui and Shaojie Wang and Wentian Zhao and Chenliang Xu},
  title = {{Improve CAM with Auto-adapted Segmentation and Co-supervised Augmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=mbaBdB846QJC},
}
Figure