Abstract
Weakly Supervised Object Localization (WSOL) methods generate both classification and localization results by learning from only image category labels. Previous methods usually utilize class activation map (CAM) to obtain target object regions. However, most of them only focus on improving foreground object parts in CAM, but ignore the important effect of its background contents. In this paper, we propose a confidence segmentation (ConfSeg) module that builds confidence score for each pixel in CAM without introducing additional hyper-parameters. The generated sample-specific confidence mask is able to indicate the extent of determination for each pixel in CAM, and further supervises additional CAM extended from internal feature maps. Besides, we introduce Co-supervised Augmentation (CoAug) module to capture feature-level representation for foreground and background parts in CAM separately. Then a metric loss is applied at batch sample level to augment distinguish ability of our model, which helps a lot to localize more related object parts. Our final model, CSoA, combines the two modules and achieves superior performance, e.g. 37.69% and 48.81% Top-1 localization error on CUB-200 and ILSVRC datasets, respectively, which outperforms all previous methods and becomes the new state-of-the-art.
Introduction
The standard tool for Weakly Supervised Object Localization is the Class Activation Map (CAM), which highlights regions a neural network uses for classification. However, CAMs often suffer from a "laziness" problem: the network identifies only the most discriminative parts of an object (e.g., a bird's beak) rather than its full extent. While previous methods attempted to solve this by "erasing" discriminative parts, they often compromised classification accuracy and ignored the background context. This research introduces CSoA (Confidence Segmentation and Co-supervised Augmentation), a framework that leverages background information as a constraint to better define foreground boundaries.
Methodology
The CSoA framework integrates two primary modules into standard CNN backbones to address the limitations of traditional CAMs:
Confidence Segmentation (ConfSeg) Module
Unlike traditional methods that use fixed thresholds, ConfSeg generates a sample-specific confidence mask. By utilizing two CAMs from different network layers ($S_L$ and $S_F$), it identifies "ambiguous" pixels (near the mean activation) and "confident" pixels (far from the mean). The confident pixels from one map supervise the refinement of the other, allowing the model to explore object-related regions without drifting into the background.
Co-supervised Augmentation (CoAug) Module
This module frames localization as a metric learning problem. It extracts feature vectors for predicted foreground and background regions and applies a loss function based on four key rules: foreground and background within an image must be distinct; foregrounds of the same category across images should be similar; foregrounds of different categories should be distinct; and backgrounds should not be misclassified as foreground.
Experimental Results
The CSoA model was evaluated on the CUB-200-2011 and ILSVRC (ImageNet) datasets, achieving significant improvements over existing state-of-the-art methods.
- CUB-200-2011: The model achieved a 37.69% Top-1 localization error, a substantial improvement over the VGG baseline (53.42%) and surpassing previous leaders like DANet (47.48%).
- ILSVRC: CSoA reached a 48.81% Top-1 localization error, outperforming methods such as ACoL and SPG.
Notably, CSoA maintains or improves classification integrity, avoiding the accuracy trade-offs common in "erasing-based" localization methods.
Conclusion
CSoA demonstrates that background context and dynamic confidence masks are powerful tools for refining object localization in weakly supervised settings. By using dual-classifier interaction and metric learning, the model captures more complete object extents. Future work will focus on making the final inference thresholding fully learnable and optimizing the framework for complex residual architectures like ResNet.
Summary updated