Contrastive Attention Maps for Self-supervised Co-localization

Minsong Ki, Youngjung Uh, Junsuk Choe, Hyeran Byun

Introduction

Object localization—the task of identifying the spatial extent of an object within an image—is a cornerstone of computer vision. Traditionally, this requires labor-intensive human annotations in the form of bounding boxes. While weakly-supervised methods have attempted to use image-level labels to reduce this burden, unsupervised co-localization represents the ultimate goal: identifying common objects in a dataset without any human-provided labels at all.

Existing self-supervised approaches often rely on "pretext tasks," such as predicting image rotation, to learn useful features. However, a significant gap remains between classification-based pretext learning and precise spatial localization. Standard attention maps generated by these models often "smear" into the background because the model only needs to identify a small, discriminative part of an object (like a bird's beak) to solve the pretext task, rather than understanding the object's full boundaries.

Proposed Methodology

To address this discrepancy, this paper introduces a framework that applies contrastive learning directly to spatial attention maps rather than global feature vectors. The architecture is built on two primary innovations:

Pixel-wise Top-k Attention Pooling (PTAP)

Standard pooling operations like max or average pooling often result in attention maps that are either too sparse or overly blurry. PTAP selectively aggregates feature map channels based on their magnitudes at each pixel. By averaging only the "top-k" most active channels, the model generates a more robust and detailed representation of the object's foreground, preventing the loss of spatial detail common in traditional pooling.

Contrastive Attention Map Loss (CAML)

This is the core contribution of the work. Instead of comparing global feature vectors, the authors define positive and negative pairs based on spatial maps to enforce consistency and background suppression:

  • Positive Pairs: The model enforces that the attention map of an original image ($A_{ori}$) matches the inversely transformed attention map of its rotated or scaled counterpart ($A_{trans2ori}$).
  • Negative Pairs: The model identifies background regions and explicitly trains the network to minimize similarity between the object's attention map and these background regions ($A_{orig2bg}$).
The proposed architecture showing the flow from input images to the Contrastive Attention Map Loss (CAML) and classification loss.

By combining these spatial constraints with a standard classification loss for the pretext task, the model is forced to maintain spatial consistency and ignore irrelevant background noise.

Experimental Results

The framework was evaluated on four fine-grained benchmarks: CUB-200-2011 (birds), Stanford Cars, FGVC-Aircraft, and Stanford Dogs. These datasets are particularly challenging for co-localization as they require distinguishing subtle differences within a single superclass.

Quantitative Performance

The proposed method achieved new state-of-the-art (SOTA) performance across all datasets. Notably, on CUB-200-2011, it reached 85.88% Correct Localization (CorLoc), surpassing the previous best of 83.78%. The improvement was even more pronounced on Stanford Dogs, where performance jumped from 73.71% to 82.82%.

Qualitative Analysis

Visualizations demonstrate that the attention maps produced by this method are significantly tighter and more accurately cover the entire object compared to previous baselines. While baseline models often focus on small parts or bleed into the background, the contrastive approach maintains focus on the object's full extent across various transformations.

Qualitative comparison between the baseline and the proposed method across different image transformations, showing more accurate bounding boxes and heatmaps.

Discussion and Limitations

While the method shows remarkable robustness across various pretext tasks (rotation, scaling, and translation), some challenges remain. The model occasionally struggles with "degenerate" cues such as strong shadows or water reflections, which it may misidentify as part of the object. Additionally, overlapping objects of the same class can lead to boundary confusion.

Conclusion

By shifting the focus of contrastive learning from global features to spatial attention maps, this research provides a powerful tool for unsupervised object co-localization. The combination of PTAP and CAML effectively bridges the gap between pretext task learning and spatial precision, setting a new standard for label-free object discovery.

Summary updated

Abstract

The goal of unsupervised co-localization is to locate the object in a scene under the assumptions that 1) the dataset consists of only one superclass, e.g., birds, and 2) there are no human-annotated labels in the dataset. The most recent method achieves impressive co-localization performance by employing self-supervised representation learning approaches such as predicting rotation. In this paper, we introduce a new contrastive objective directly on the attention maps to enhance co-localization performance. Our contrastive loss function exploits rich information of location, which induces the model to activate the extent of the object effectively. In addition, we propose a pixel-wise attention pooling that selectively aggregates the feature map regarding their magnitudes across channels. Our methods are simple and shown effective by extensive qualitative and quantitative evaluation, achieving state-of-the-art co-localization performances by large margins on four datasets: CUB-200-2011, Stanford Cars, FGVC-Aircraft, and Stanford Dogs. Our code will be publicly available online for the research community.
BibTeX
@inproceedings{ki2021contrastive,
  author = {Minsong Ki and Youngjung Uh and Junsuk Choe and Hyeran Byun},
  title = {{Contrastive Attention Maps for Self-supervised Co-localization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00280},
}
Figure