Combinational Class Activation Maps for Weakly Supervised Object Localization

Seunghan Yang, Yoonhyung Kim, Youngeun Kim, Changick Kim

Abstract

Weakly supervised object localization (WSOL) has recently attracted attention since it aims to identify both class labels and locations of objects by using image-level labels. Most previous methods utilize the activation map corresponding to the highest activation source. Exploiting only one activation map of the highest probability class is often biased into limited regions or sometimes even highlights background regions. To resolve these limitations, we propose to use activation maps, named combinational class activation maps (CCAM), which are linear combinations of activation maps from the highest to the lowest probability class. By using CCAM for localization, we suppress background regions to help highlighting foreground objects more accurately. In addition, we design the network architecture to consider spatial relationships for localizing relevant object regions. Specifically, we integrate non-local modules into an existing base network at both low- and high-level layers. Our final model, named non-local combinational class activation maps (NL-CCAM), obtains superior performance compared to previous methods on representative object localization benchmarks including ILSVRC 2016 and CUB-200-2011.

Introduction

Weakly Supervised Object Localization (WSOL) is a cost-effective alternative to standard object detection. Instead of requiring labor-intensive bounding box annotations, WSOL models are trained using only image-level labels (e.g., "bird," "car"). The standard baseline for this task is Class Activation Mapping (CAM), which identifies the pixels that most contribute to a classifier's prediction.

However, CAM suffers from two primary limitations: discriminative bias and background noise. Discriminative bias occurs when a model focuses only on the most distinct part of an object (like a bird's beak) rather than its full extent. Background noise arises when the model incorrectly activates on elements that frequently co-occur with the target object, such as water appearing with boats or grass with animals.

Methodology

Combinational Class Activation Maps (CCAM)

To address background noise, the authors introduce CCAM. While traditional CAM only utilizes the activation map for the highest-probability class, CCAM leverages the insight that low-probability class maps often contain useful information about what the network considers "background."

Overview of the CCAM process, showing how activation maps from high to low probability classes are combined to produce the final localization map.

By creating a linear combination of maps from the highest to the lowest probability classes, the model can effectively "subtract" background regions. The researchers explored various weighting functions, including polynomial curves and a "Top-i & Bottom-j" approach, to find the optimal balance for suppressing non-object regions while preserving the foreground.

Non-Local Modules

To combat discriminative bias, the architecture integrates non-local modules into a VGG16 backbone. Unlike standard convolutions that operate locally, non-local modules compute relationships between all pixels in a feature map. By applying these modules at both low- and high-level layers, the network can capture long-range spatial dependencies. This allows the model to recognize that disparate parts of an object (e.g., a bird's head and tail) are spatially related, leading to more complete and accurate bounding boxes.

Experimental Results

The proposed NL-CCAM was evaluated on the ILSVRC 2016 (ImageNet) and CUB-200-2011 datasets, achieving state-of-the-art results. On ImageNet, it achieved a Top-1 localization error of 49.83%, a significant improvement over the original CAM (57.20%). On the fine-grained CUB-200-2011 dataset, it reduced the Top-1 localization error to 47.60%.

Qualitative comparison between standard CAM and the proposed NL-CCAM across different object categories, demonstrating superior background suppression and object coverage.

Qualitative analysis (as seen in the figure above) demonstrates that the combination of multiple class maps ($M^{c_1}$ through $M^{c_K}$) allows the model to suppress background clutter that often confuses standard CAM. For instance, in the lighthouse example, standard CAM activates on the surrounding trees and sky, whereas NL-CCAM tightly localizes the structure itself.

Conclusion

NL-CCAM demonstrates that the "wrong" class predictions in a classifier are not merely failures but valuable spatial indicators of the background. By combining these maps and utilizing non-local modules for better spatial context, the method provides a robust, single-forward-pass solution for weakly supervised localization. Future work may involve adaptive weighting mechanisms to further refine how these maps are combined for diverse image types.

Summary updated

Abstract

Weakly supervised object localization has recently attracted attention since it aims to identify both class labels and locations of objects by using image-level labels. Most previous methods utilize the activation map corresponding to the highest activation source. Exploiting only one activation map of the highest probability class is often biased into limited regions or sometimes even highlights background regions. To resolve these limitations, we propose to use activation maps, named combinational class activation maps (CCAM), which are linear combinations of activation maps from the highest to the lowest probability class. By using CCAM for localization, we suppress background regions to help highlighting foreground objects more accurately. In addition, we design the network architecture to consider spatial relationships for localizing relevant object regions. Specifically, we integrate non-local modules into an existing base network at both low- and high-level layers. Our final model, named non-local combinational class activation maps (NL-CCAM), obtains superior performance compared to previous methods on representative object localization benchmarks including ILSVRC 2016 and CUB- 200-2011. Furthermore, we show that the proposed method has a great capability of generalization by visualizing other datasets.
BibTeX
@inproceedings{yang2020combinational,
  author = {Seunghan Yang and Yoonhyung Kim and Youngeun Kim and Changick Kim},
  title = {{Combinational Class Activation Maps for Weakly Supervised Object Localization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=dpLmRL-BZ8},
}
Figure