Abstract
Distance Metric Learning (DML) involves learning an embedding that brings similar examples closer while moving away dissimilar ones. Existing DML approaches make use of class labels to generate constraints for metric learning. In this paper, we address the less-studied problem of learning a metric in an unsupervised manner. We do not make use of class labels, but use unlabeled data to generate adversarial, synthetic constraints for learning a metric inducing embedding. Being a measure of uncertainty, we minimize the entropy of a conditional probability to learn the metric. Our stochastic formulation scales well to large datasets, and performs competitive to existing metric learning methods.
Introduction
Distance Metric Learning (DML) is a fundamental task in machine learning, aimed at teaching models to map data into a feature space where "similar" items are clustered together and "dissimilar" items are pushed apart. Traditionally, this process relies heavily on large-scale labeled datasets to define these relationships (e.g., identifying that two images both belong to the same class). However, in many domains such as medical imaging or large-scale web crawls, obtaining human-annotated labels is often prohibitively expensive or impossible.
Existing unsupervised DML methods frequently rely on manifold learning, which captures the geometric structure of data but often fails to grasp deeper semantic relationships. This paper introduces SUML (Synthetic Unsupervised Metric Learning), a framework designed to bridge this gap by learning a semantic-aware metric without any human-provided labels.
Methodology
The core of SUML is an "imagine and compete" strategy that generates synthetic constraints to guide the learning process. The framework operates through several key stages:
- Hypothetical Labeling: The model samples triplets of unlabeled data $(x_a, x_p, x_n)$ and considers five hypothetical semantic cases ($c=1$ to $5$). These cases represent different potential relationships, such as whether the anchor and positive share a class or if all three points are distinct.
- Synthetic Example Generation: For each hypothetical case, a learned mapping $S$ transforms the original data into "synthetic" anchors, positives, and negatives. This mapping is trained adversarially; it attempts to create synthetic examples that are intentionally difficult for the metric to resolve.
- Entropy Minimization: The model treats the likelihood of each hypothetical case as a probability distribution. By minimizing the entropy of this distribution, the model reduces uncertainty, forcing the metric to converge on the most plausible semantic relationships for the data structure.
To handle large-scale datasets like ImageNet, the authors propose sSUML, which utilizes Riemannian product manifold optimization to jointly learn the feature extractor and the metric in a scalable, stochastic manner.
Experimental Results
The authors evaluated SUML across a diverse range of benchmarks, including small-scale classification (AT&T Faces, USPS), Zero-Shot Learning (HMDB), and large-scale visual categorization (ImageNet, CUB-200, SOP).
Performance and Efficiency
SUML demonstrates performance competitive with supervised state-of-the-art (SOTA) methods like JDRML and LRGMML, despite the total absence of labels. It consistently outperforms classical unsupervised baselines such as NPE and LPP. Notably, SUML is highly efficient; on the HMDB dataset, it achieved training in just 1.38 seconds, compared to 864 seconds for the supervised NCA method.
Sensitivity and Robustness
Analysis of the model's hyperparameters shows that while the performance is relatively stable across different values of the angular constraint $\alpha$, it is more sensitive to the trade-off parameter $\lambda$, which governs the orthogonality constraint. Proper tuning of $\lambda$ is essential to prevent model collapse. Additionally, the model proved robust in noisy environments, maintaining high accuracy even when 20-25% of the dataset consisted of irrelevant "noise" data.
Conclusion
SUML provides a robust framework for unsupervised metric learning by leveraging adversarial synthetic examples and entropy minimization. By creating difficult constraints rather than searching for them, the model learns a more resilient feature space. Future research may explore more complex synthetic generation functions to further enhance the quality of the learned semantic embeddings.
Summary updated