Abstract
Existing approaches for unsupervised metric learning focus on exploring self-supervision information within the input image itself. We observe that, when analyzing images, human eyes often compare images against each other instead of examining images individually. In addition, they often pay attention to certain keypoints, image regions, or objects which are discriminative between image classes but highly consistent within classes. Even if the image is being transformed, the attention pattern will be consistent. Motivated by this observation, we develop a new approach to unsupervised deep metric learning where the network is learned based on self-supervision information across images instead of within one single image. To characterize the consistent pattern of human attention during image comparisons, we introduce the idea of transformed attention consistency. It assumes that visually similar images, even undergoing different image transforms, should share the same consistent visual attention map. This consistency leads to a pairwise self-supervision loss, allowing us to learn a Siamese deep neural network to encode and compare images against their transformed or matched pairs. To further enhance the inter-class discriminative power of the feature generated by this network, we adapt the concept of triplet loss from supervised metric learning to our unsupervised case and introduce the contrastive clustering loss. Our extensive experimental results on benchmark datasets demonstrate that our proposed method outperforms current state-of-the-art methods for unsupervised metric learning by a large margin.
Introduction
Deep Metric Learning (DML) aims to learn a mapping from image space to a feature space where semantically similar images are clustered together. While supervised DML has seen significant success, it relies on expensive labeled datasets. Unsupervised DML seeks to overcome this by leveraging self-supervision. However, most existing methods focus on intra-image consistency (e.g., an image and its augmented version). This paper argues that such approaches ignore the human ability to learn through cross-image comparison—identifying consistent features across different instances of a category.
Methodology: The TAC-CCL Framework
The proposed framework, TAC-CCL, introduces two primary mechanisms for unsupervised learning: Transformed Attention Consistency (TAC) and Contrastive Clustering Loss (CCL).
Transformed Attention Consistency (TAC)
Humans often focus on specific discriminative regions (e.g., a bird's beak or a car's wheel) that remain consistent across different views or transformations. To mimic this, the authors integrate a Convolutional Block Attention Module (CBAM) into the network. The TAC loss enforces that the attention map generated for an image $x_n$ remains consistent when the image undergoes a spatial transformation $T$ to become $x'_n$.
As shown in the figure above, the attention maps for birds and cars remain focused on the same semantic parts even after the input images are transformed. This consistency provides a powerful self-supervisory signal, ensuring the model learns features that are robust to geometric changes.
Framework Overview
The overall architecture utilizes a Siamese network structure with a shared backbone $G$. The system is designed to process pairs of images, which can be an image and its transform, or matched pairs identified across different images in the dataset.
The pipeline processes these pairs to compute the TAC loss ($L_{TAC}$) and the Contrastive Clustering Loss ($L_{CC}$). A memory bank is employed to store features across the entire dataset, enabling comparisons beyond the current mini-batch and providing a global context for the clustering process.
Contrastive Clustering Loss (CCL)
To enhance inter-class discriminative power without labels, the authors use $k$-means clustering on the feature space to generate pseudo-labels. The CCL then applies a triplet-like constraint: a feature should be significantly closer to its assigned cluster center than to the next nearest cluster center. This "contrastive ratio" effectively pushes different clusters apart in the embedding space, mimicking the discriminative power of supervised learning.
Cross-Image Supervision via Affine-SIFT
A key innovation is the use of Affine-SIFT to find matching keypoints between different images in the dataset. This allows the model to identify similar objects across different scenes, providing a "cross-image" signal that is much richer than standard single-image data augmentation. This allows the network to learn semantic features (like "car wheels" or "bird wings") rather than just memorizing specific pixel patterns.
Experimental Results
The method was evaluated on three benchmark datasets: CUB-200-2011 (birds), Cars-196 (cars), and Stanford Online Products (SOP). The proposed TAC-CCL method achieved state-of-the-art (SOTA) results across the board:
- CUB: Improved Recall@1 by 11.3% over the previous best unsupervised method, reaching 57.5%.
- Cars: Outperformed the previous SOTA by 4.8% (Recall@1 of 46.1%).
- SOP: Achieved a massive 15.0% improvement in Recall@1 (jumping from 48.9% to 63.9%).
Qualitative analysis via t-SNE visualizations confirms that the learned feature space exhibits well-defined and separated clusters compared to baseline unsupervised methods.
Conclusion
By shifting the focus from individual image consistency to cross-image attention consistency and leveraging contrastive clustering, this work sets a new state-of-the-art for unsupervised deep metric learning. The integration of attention as a training constraint and the use of classical computer vision techniques like SIFT for self-supervision prove to be highly effective strategies for learning discriminative embeddings without manual labels.
Summary updated