Category-aware Allocation Transformer for Weakly Supervised Object Localization

Zhiwei Chen, Jinren Ding, Liujuan Cao, Yunhang Shen, Shengchuan Zhang, Guannan Jiang, Rongrong Ji

Introduction

Weakly Supervised Object Localization (WSOL) aims to localize objects within an image using only image-level category labels during training. This task is challenging because the model must infer spatial extents without explicit bounding box supervision. Traditional methods relying on Class Activation Maps (CAMs) often focus on the most discriminative parts of an object (e.g., a bird's head) rather than its full extent. While Vision Transformers (ViTs) offer long-range dependencies that help capture global context, their self-attention maps are typically category-agnostic, leading to noisy localization that fails to distinguish between different classes in a scene.

Comparison of localization mechanisms: (A) CAM-based, (B) standard Transformer-based (category-agnostic), and (C) the proposed Category-aware mechanism.

Methodology: The CATR Framework

The proposed Category-aware Allocation Transformer (CATR) addresses these limitations by forcing the transformer's internal attention mechanisms to become class-specific. The architecture introduces two primary components: the Category-aware Stimulation Module (CSM) and the Object Constraint Module (OCM).

The CATR architecture, highlighting the Category-aware Stimulation Module (CSM) and the Object Constraint Module (OCM) integrated with a Transformer backbone.

Category-aware Stimulation Module (CSM)

The CSM is designed to "stimulate" the transformer's attention maps to learn category-specific features. It takes raw attention maps and processes them through a CNN-based encoder-decoder structure. By applying an auxiliary classification loss ($L_{csm}$) to these maps, the model is incentivized to highlight regions that are not just "objects" but are specifically relevant to the target class.

Object Constraint Module (OCM)

The OCM refines the localization by addressing two common issues: background noise and incomplete object coverage.

  • Noise Suppression: A percentile-based thresholding technique is used to filter out low-activation regions, effectively silencing background clutter.
  • Object Awakening: This mechanism aggregates features across multiple transformer blocks to generate pixel-level pseudo-labels. These labels provide a self-supervised signal that helps the model "wake up" the full geometry of the object, ensuring that non-discriminative but relevant parts (like a bird's tail) are included in the final localization map.

To ensure stable training, CATR employs a learnable weighting mechanism that automatically balances the primary classification loss with the auxiliary losses from the CSM and OCM.

Experiments and Results

The CATR model was evaluated on two major benchmarks: CUB-200-2011 (fine-grained bird species) and ILSVRC (ImageNet).

Quantitative Performance

CATR achieved state-of-the-art results across both datasets:

  • CUB-200-2011: The model reached a 79.62% Top-1 Localization Accuracy, significantly outperforming previous transformer-based methods like LCTR.
  • ILSVRC: CATR achieved 56.90% Top-1 Localization Accuracy, a substantial improvement over the TS-CAM baseline (53.40%).

Qualitative Analysis

Visualizations demonstrate that CATR produces more comprehensive localization maps compared to previous methods. While standard CAMs often highlight only small, highly discriminative regions, CATR's category-aware attention maps consistently cover the entire body of the target object with tighter boundaries and less background interference.

Conclusion

By integrating category awareness into the transformer's self-attention mechanism, CATR effectively bridges the gap between global feature dependency and class-specific localization. The combination of the Category-aware Stimulation Module and the Object Constraint Module allows the model to suppress noise and capture the full extent of objects, setting a new benchmark for weakly supervised object localization.

Summary updated

Abstract

Weakly supervised object localization (WSOL) aims to localize objects based on only image-level labels as supervision. Recently, transformers have been introduced into WSOL, yielding impressive results. The self-attention mechanism and multilayer perceptron structure in transformers preserve long-range feature dependency, facilitating complete localization of the full object extent. However, current transformer-based methods predict bounding boxes using category-agnostic attention maps, which may lead to confused and noisy object localization. To address this issue, we propose a novel Category-aware Allocation TRansformer (CATR) that learns category-aware representations for specific objects and produces corresponding category-aware attention maps for object localization. First, we introduce a Category-aware Stimulation Module (CSM) to induce learnable category biases for self-attention maps, providing auxiliary supervision to guide the learning of more effective transformer representations. Second, we design an Object Constraint Module (OCM) to refine the object regions for the category-aware attention maps in a self-supervised manner. Extensive experiments on the CUB-200-2011 and ILSVRC datasets demonstrate that the proposed CATR achieves significant and consistent performance improvements over competing approaches.
BibTeX
@inproceedings{chen2023category,
  author = {Zhiwei Chen and Jinren Ding and Liujuan Cao and Yunhang Shen and Shengchuan Zhang and Guannan Jiang and Rongrong Ji},
  title = {{Category-aware Allocation Transformer for Weakly Supervised Object Localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00611},
}
Figure