Integral Object Mining via Online Attention Accumulation

Peng-Tao Jiang, Qibin Hou, Yang Cao, Ming-Ming Cheng, Yunchao Wei, Hongkai Xiong

Introduction

Semantic segmentation—the task of assigning a category label to every pixel in an image—is a cornerstone of computer vision but remains labor-intensive due to the need for precise, hand-drawn masks. Weakly-Supervised Semantic Segmentation (WSSS) aims to mitigate this by learning from image-level tags (e.g., "cat" or "bus").

The standard approach for WSSS utilizes Class Activation Maps (CAMs). However, CAMs typically suffer from "tunnel vision": a classifier only needs to identify the most discriminative part of an object (like a bird's beak) to classify the image correctly. This results in sparse attention maps that fail to cover the entire object, providing a poor foundation for full-object segmentation.

Methodology

This paper introduces a two-stage framework to overcome the limitations of sparse CAMs by leveraging the temporal dynamics of the training process.

Online Attention Accumulation (OAA)

The authors observed that while a network's attention is sparse at any single moment, its focus shifts across different training epochs. In one epoch, it might focus on a dog's face; in another, its paws. OAA maintains a cumulative attention map $M$ for every image. At each epoch $t$, the new attention map $A_t$ is merged with the previous cumulative map $M_{t-1}$ using an element-wise maximum operation: $$M_t = \max(M_{t-1}, A_t)$$ This allows the network to progressively "paint in" the entire object. To ensure quality, updates only occur when the classification confidence for the target category is high.

Overview of the OAA and Integral Attention Learning framework.

Integral Attention Learning (OAA+)

While accumulated maps are more complete, they may still contain low-confidence regions. The OAA+ stage treats these cumulative maps as soft labels to train a dedicated "Integral Attention Model." This model is optimized using a Hybrid Loss:

  • Enhanced Loss: Encourages higher predictions for object regions, even those with low initial confidence.
  • Constraint Loss: Penalizes the network for expanding into known background areas, preventing over-segmentation.

Experimental Results

The proposed method was evaluated on the PASCAL VOC 2012 benchmark, using a ResNet-101 backbone.

Quantitative Performance

OAA+ achieved state-of-the-art results for image-level supervised segmentation, reaching a 66.4% mIoU on the test set. This significantly outperforms previous leading methods such as SeeNet (60.7%) and AE-PSL (55.7%). Notably, the model is highly data-efficient; when trained on only 20% of the dataset, OAA+ still achieved 54.6% mIoU, surpassing several older methods trained on the full dataset.

Qualitative Analysis

Visual comparisons demonstrate that OAA and OAA+ produce significantly more complete and "thicker" object masks than standard CAMs. By accumulating shifting focal points, the model successfully captures the full extent of objects across diverse categories.

Qualitative comparison of attention maps generated by CAM, OAA, and OAA+ across various object classes.

Conclusion

Integral Object Mining via OAA offers a simple, "plug-and-play" strategy for improving WSSS. By treating training as a temporal process, the method harvests far more information from standard classifiers than static approaches. While the reliance on external saliency maps for background cues remains a limitation, OAA provides a robust path toward more efficient and accurate weakly-supervised segmentation.

Summary updated

Abstract

Object attention maps generated by image classifiers are usually used as priors for weakly-supervised segmentation approaches. However, normal image classifiers produce attention only at the most discriminative object parts, which limits the performance of weakly-supervised segmentation task. Therefore, how to effectively identify entire object regions in a weakly-supervised manner has always been a challenging and meaningful problem. We observe that the attention maps produced by a classification network continuously focus on different object parts during training. In order to accumulate the discovered different object parts, we propose an online attention accumulation (OAA) strategy which maintains a cumulative attention map for each target category in each training image so that the integral object regions can be gradually promoted as the training goes. These cumulative attention maps, in turn, serve as the pixel-level supervision, which can further assist the network in discovering more integral object regions. Our method (OAA) can be plugged into any classification network and progressively accumulate the discriminative regions into integral objects as the training process goes. Despite its simplicity, when applying the resulting attention maps to the weakly-supervised semantic segmentation task, our approach improves the existing state-of-the-art methods on the PASCAL VOC 2012 segmentation benchmark, achieving a mIoU score of 66.4% on the test set. Code is available at https://mmcheng.net/oaa/.
BibTeX
@inproceedings{jiang2019integral,
  author = {Peng-Tao Jiang and Qibin Hou and Yang Cao and Ming-Ming Cheng and Yunchao Wei and Hongkai Xiong},
  title = {{Integral Object Mining via Online Attention Accumulation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00216},
}
Figure