Unsupervised Deep Metric Learning with Transformed Attention Consistency and Contrastive Clustering Loss

Yang Li, Shichao Kan, Zhihai He

Abstract

Existing approaches for unsupervised metric learning focus on exploring self-supervision information within the input image itself. We observe that, when analyzing images, human eyes often compare images against each other instead of examining images individually. In addition, they often pay attention to certain keypoints, image regions, or objects which are discriminative between image classes but highly consistent within classes. Even if the image is being transformed, the attention pattern will be consistent. Motivated by this observation, we develop a new approach to unsupervised deep metric learning where the network is learned based on self-supervision information across images instead of within one single image. To characterize the consistent pattern of human attention during image comparisons, we introduce the idea of transformed attention consistency. It assumes that visually similar images, even undergoing different image transforms, should share the same consistent visual attention map. This consistency leads to a pairwise self-supervision loss, allowing us to learn a Siamese deep neural network to encode and compare images against their transformed or matched pairs. To further enhance the inter-class discriminative power of the feature generated by this network, we adapt the concept of triplet loss from supervised metric learning to our unsupervised case and introduce the contrastive clustering loss. Our extensive experimental results on benchmark datasets demonstrate that our proposed method outperforms current state-of-the-art methods for unsupervised metric learning by a large margin.

Introduction

Deep Metric Learning (DML) aims to learn a feature embedding where similar images are mapped to nearby points and dissimilar ones are pushed apart. While supervised DML has seen great success, it relies on extensive manual labeling. Unsupervised DML seeks to learn these embeddings without labels, typically by leveraging self-supervision from individual images and their augmentations.

However, human visual learning is inherently comparative. We identify discriminative features—like the specific shape of a bird's beak—by comparing different instances. This paper proposes a framework that moves beyond single-image self-supervision by introducing Transformed Attention Consistency (TAC) and Contrastive Clustering Loss (CCL). By enforcing consistency in where a network "looks" across transformations and different but similar images, the model learns robust, class-level features.

Methodology

The proposed framework utilizes a Siamese network architecture to process image pairs and learn discriminative embeddings through two primary mechanisms.

Transformed Attention Consistency (TAC)

The network incorporates an attention mechanism to generate attention maps that highlight salient regions. The TAC loss ensures that if an image $x_n$ is transformed into $x'_n$ (via rotation, cropping, etc.), the corresponding attention map should undergo the same transformation.

Crucially, this consistency is extended across different images. By using affine-SIFT to identify matching keypoints between distinct images in the dataset, the model identifies "matched" pairs. Enforcing attention consistency across these pairs allows the network to discover semantic features common to a class without explicit labels.

The TAC-CCL framework architecture, showing the Siamese network, attention maps, and the integration of TAC and CCL losses.

Contrastive Clustering Loss (CCL)

To provide inter-class discriminative power, the authors introduce CCL, an unsupervised adaptation of the triplet loss. A Memory Bank stores features from the entire dataset, providing a global view of the feature distribution. The model performs $k$-means clustering on these stored features. The CCL then encourages each feature to be significantly closer to its assigned cluster center than to any other center, promoting compact and well-separated clusters in the embedding space.

Examples of cross-image matching using affine-SIFT (d, e) and resulting unsupervised clusters (a, b, c).

Experiments and Results

The method was evaluated on three standard DML benchmarks: CUB-200-2011 (birds), Cars-196 (cars), and Stanford Online Products (SOP). Following standard protocols, the model was trained on the first half of the classes and tested on the remaining unseen classes.

The TAC-CCL method achieved significant improvements over previous unsupervised state-of-the-art (SOTA) methods:

  • CUB: Recall@1 reached 57.5%, an 11.3% absolute improvement over the previous best unsupervised method.
  • SOP: On this large-scale dataset, the method achieved 63.9% Recall@1, outperforming the previous SOTA by 15%.
  • Ablation Studies: Results confirmed that while clustering and memory banks provide a strong baseline, the addition of TAC consistently improves performance by 2-4%, highlighting the importance of attention consistency.

Conclusion

This work demonstrates that mimicking human comparative visual behavior through attention consistency and global clustering can significantly bridge the gap between unsupervised and supervised deep metric learning. While the reliance on pre-processed keypoint matching and sensitivity to the number of clusters ($K$) present areas for future optimization, the TAC-CCL framework sets a new benchmark for unsupervised feature representation.

Summary updated

Abstract

Existing approaches for unsupervised metric learning focus on exploring self-supervision information within the input image itself. We observe that, when analyzing images, human eyes often compare images against each other instead of examining images individually. In addition, they often pay attention to certain keypoints, image regions, or objects which are discriminative between image classes but highly consistent within classes. Even if the image is being transformed, the attention pattern will be consistent. Motivated by this observation, we develop a new approach to unsupervised deep metric learning where the network is learned based on self-supervision information across images instead of within one single image. To characterize the consistent pattern of human attention during image comparisons, we introduce the idea of transformed attention consistency. It assumes that visually similar images, even undergoing different image transforms, should share the same consistent visual attention map. This consistency leads to a pairwise self-supervision loss, allowing us to learn a Siamese deep neural network to encode and compare images against their transformed or matched pairs. To further enhance the inter-class discriminative power of the feature generated by this network, we adapt the concept of triplet loss from supervised metric learning to our unsupervised case and introduce the contrastive clustering loss. Our extensive experimental results on benchmark datasets demonstrate that our proposed method outperforms current state-of-the-art methods for unsupervised metric learning by a large margin.
BibTeX
@inproceedings{li2020unsupervised,
  author = {Yang Li and Shichao Kan and Zhihai He},
  title = {{Unsupervised Deep Metric Learning with Transformed Attention Consistency and Contrastive Clustering Loss}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=PfdP7gM6CMu},
}
Figure