Uncertainty-Guided Metric Learning without Labels

Dhanunjaya Varma Devalraju, C Chandra SekharWACV 2025

TL;DR

This paper introduces an "Uncertainty-Guided Metric Learning" (UGML) framework to improve unsupervised learning of image similarity. By leveraging contextual information and the uncertainty of model predictions, UGML refines noisy pseudo-labels generated from clustering, leading to more discriminative embeddings and better performance in tasks like image retrieval, while also being computationally more efficient than some existing methods.

What Problem Does This Paper Address?

Learning to understand similarity between objects is fundamental in computer vision. While supervised metric learning relies on explicit labels, unsupervised methods aim to learn these similarities from unlabeled data. A common approach in unsupervised metric learning is "pseudo-labeling," where clustering is used to generate temporary labels for the data. However, these pseudo-labels are often noisy, which can significantly degrade the performance of the learned models. Existing methods try to mitigate this noise using auxiliary tasks or by refining pseudo-labels, but the problem of noisy labels and its impact on learning discriminative representations persists. This paper tackles the challenge of learning robust and discriminative embeddings in the absence of labels, specifically by addressing the inherent noise in pseudo-labels generated by clustering.

Methods

The UGML framework consists of four main steps:

  1. Pseudo-label Generation: A pre-trained CNN (feature extractor) is used to generate initial features from unlabeled data. These features are then clustered (e.g., using k-means) to produce initial pseudo-labels.
  2. Classifier Training for Uncertainty Estimation: A classification network is trained using these noisy pseudo-labels. This network is designed to estimate prediction uncertainty. To achieve this, the authors employ Monte Carlo (MC) dropout during inference, allowing for multiple stochastic forward passes to generate a distribution of predictions.
  3. Uncertainty Estimation using MC Dropout and Contextual Similarity: The uncertainty from the MC dropout is combined with contextual information. To incorporate context, the framework aggregates information from k-nearest neighbors in the feature space. This neighborhood aggregation helps refine the model's predictions and uncertainty estimates, making them more reliable, especially when initial pseudo-labels are noisy. The refined predictions are used to compute confidence scores (prediction mean) and uncertainty (prediction variance) for each example.
  4. Training the Embedding Network: The core of the learning happens here. The authors modify the multi-similarity loss function to incorporate the computed confidence and uncertainty. A pairwise weight term is introduced, which assigns lower weights to pairs with high uncertainty and low confidence, effectively downplaying noisy or unreliable pseudo-labels. This weighted loss function is then used to train the final embedding network.

Datasets

The proposed framework is evaluated on three standard benchmark datasets for metric learning:

  • CUB-200-2011 (CUB): A dataset of bird species.
  • Cars-196 (Cars): A dataset of car models.
  • Stanford Online Product (SOP): A dataset of product images.

The paper also mentions using ImageNet pre-trained models for feature extraction and classification.

Key Insights and Ideas

  • Leveraging Uncertainty for Noise Reduction: The core idea is that model uncertainty is a strong indicator of label noise. By quantifying and using this uncertainty, the framework can intelligently down-weight or ignore unreliable pseudo-labels.
  • Contextual Refinement of Predictions: Relying solely on a classifier's prediction can be insufficient when labels are noisy. Aggregating information from semantically similar neighbors (context) significantly improves the reliability of predictions and uncertainty estimates.
  • Pairwise Weighting for Robust Loss: Directly modifying the loss function with per-example weights is challenging. The paper ingeniously derives a pairwise weight that incorporates the confidence and uncertainty of the involved examples, making the metric learning loss more robust to noisy pseudo-labels.
  • Computational Efficiency: The proposed method is shown to be computationally more efficient than some state-of-the-art methods like STML, which involve more complex iterative refinement processes.

Key Results

  • The proposed UGML framework achieves performance comparable to state-of-the-art methods on CUB, Cars, and SOP datasets.
  • On the Cars dataset with 512-dimensional embeddings and R-MAC features, UGML outperforms all other techniques.
  • It achieves competitive results across various embedding dimensions and feature extraction methods (SCDA and R-MAC).
  • When trained from scratch on SOP with a randomly initialized ResNet-18, UGML shows strong performance, outperforming other methods by a significant margin in Recall@1, Recall@10, and Recall@100.
  • The ablation studies confirm that both pseudo-label refinement (neighborhood aggregation) and uncertainty-based pair weighting contribute to performance gains, with their combination yielding the best results.
  • UGML is computationally more efficient than STML, requiring less training time.

Limitations and Future Work

  • Dependency on Initial Features: The performance of UGML is limited by the quality of the initial features extracted by the pre-trained CNN. The paper acknowledges that if these initial features are poor, the subsequent steps might not yield optimal results.
  • No Pseudo-label Update: The current framework refines pseudo-labels once and does not iteratively update them based on the learned embedding. This differs from methods like STML, UHML, and ROUL, which might offer further improvements through iterative refinement.
  • Future Work: The authors suggest extending the approach to reduce its dependency on the quality of initial features. This could involve exploring more advanced feature extraction techniques or methods that can adaptively learn better features during the unsupervised metric learning process.

Summary updated

BibTeX
@inproceedings{devalraju2025uncertainty,
  author = {Dhanunjaya Varma Devalraju and C Chandra Sekhar},
  title = {{Uncertainty-Guided Metric Learning without Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure