Abstract
In unsupervised learning of image features without labels, especially on datasets with fine-grained object classes, it is often very difficult to tell if a given image belongs to one specific object class or another, even for human eyes. However, we can reliably tell if image C is more similar to image A than image B. In this work, we propose to explore how this relative order can be used to learn discriminative features with an unsupervised metric learning method. Instead of resorting to clustering or self-supervision to create pseudo labels for an absolute decision, which often suffers from high label error rates, we construct reliable relative orders for groups of image samples and learn a deep neural network to predict these relative orders. During training, this relative order prediction network and the feature embedding network are tightly coupled, providing mutual constraints to each other to improve overall metric learning performance in a cooperative manner. During testing, the predicted relative orders are used as constraints to optimize the generated features and refine their feature distance-based image retrieval results using a constrained optimization procedure. Our experimental results demonstrate that the proposed relative orders for unsupervised learning (ROUL) method is able to significantly improve the performance of unsupervised deep metric learning.
Introduction: The Challenge of Unsupervised Metric Learning
Deep metric learning (DML) aims to map images into a feature space where distances reflect semantic similarity. In unsupervised settings, the lack of ground-truth labels typically forces a reliance on clustering-based pseudo-labels or self-supervised pretext tasks. However, these methods often struggle with "fine-grained" datasets—where differences between classes are subtle—leading to high label noise, sometimes reaching error rates of 40-60%.
The authors of this paper observe that while absolute categorization is difficult even for humans in these contexts, relative similarity is much more intuitive. It is significantly easier to decide if image C is more similar to image A than it is to image B, rather than assigning each to a specific species without a reference.
Methodology: Relative Orders for Unsupervised Learning (ROUL)
The ROUL framework introduces a novel approach by training a network to predict these relative orders. The system comprises two main components:
- Feature Embedding Network ($F_\gamma$): Responsible for mapping images into the discriminative feature space.
- Relative Order Prediction Network ($\Phi_\theta$): Predicts a "Relative Order Matrix" to determine the similarity ranking of image triplets.
Reliable Order Construction
To avoid the pitfalls of noisy labels, ROUL constructs high-confidence relative orders using data augmentation and cluster proximity. An image is naturally more similar to its own augmented versions than to images from distant clusters. By defining a ternary relative order function that includes an "uncertain" (0) state, the model ignores ambiguous pairs and focuses on high-confidence signals.
Cooperative Training and Test-Time Optimization
The two networks are trained with mutual constraints. The Relative Order Consistency loss ensures the embedding network respects the predicted orders, while the Metric Order Consistency loss aligns the order network with the current feature distances.
Uniquely, ROUL also employs test-time optimization. During inference, the predicted relative orders act as constraints to refine the feature distances, effectively re-ranking retrieval results to suppress false positives that may appear visually similar but belong to different classes.
Experimental Results
The method was evaluated on three standard benchmarks: CUB-200-2011 (birds), Cars-196, and Stanford Online Products (SOP).
- State-of-the-Art Performance: ROUL significantly outperformed existing unsupervised methods. On the CUB-200-2011 dataset, it achieved a Recall@1 of 56.7%, marking an 8.6% improvement over the previous best method (PSLR).
- Scalability and Robustness: On the large-scale SOP dataset, ROUL improved Recall@1 by 2.3%. Even when trained from scratch (without ImageNet pre-training), it maintained a 3.1% lead over competitors.
- Qualitative Gains: Visualizations of retrieval results confirmed that the test-time optimization successfully pushes visually similar but semantically different images further down the ranking, improving the precision of the top results.
Conclusion
ROUL demonstrates that relative similarity is a powerful and stable signal for unsupervised deep metric learning. By moving away from error-prone absolute pseudo-labels and embracing a cooperative learning framework with test-time refinement, the method sets a new standard for learning discriminative features without human supervision.
Summary updated