Unsupervised Embedding Quality Evaluation

Anton Tsitsulin, Marina Munkhoeva, Bryan Perozzi

Abstract

Unsupervised learning has recently significantly gained in popularity, especially with deep learning-based approaches. Despite numerous successes and approaching supervised-level performance on a variety of academic benchmarks, it is still hard to train and evaluate SSL models in practice due to the unsupervised nature of the problem. Even with networks trained in a supervised fashion, it is often unclear whether they will perform well when transferred to another domain. Past works are generally limited to assessing the amount of information contained in embeddings, which is most relevant for self-supervised learning of deep neural networks. This work chooses to follow a different approach: can we quantify how easy it is to linearly separate the data in a stable way? We survey the literature and uncover three methods that could be potentially used for evaluating quality of representations. We also introduce one novel method based on recent advances in understanding the high-dimensional geometric structure self-supervised learning. We conduct extensive experiments and study the properties of these metrics and ones introduced in the previous work. Our results suggest that while there is no free lunch, there are metrics that can robustly estimate embedding quality in an unsupervised way.

Introduction

In Self-Supervised Learning (SSL), models are trained to generate mathematical representations, or embeddings, of data without explicit labels. A significant challenge in this field is evaluating the quality of these embeddings. The current standard involves training a "linear probe"—a supervised classifier—on top of the frozen embeddings. This approach is paradoxical: if sufficient labels are available for evaluation, the need for unsupervised learning is diminished. Moreover, embeddings that perform well on one dataset may not transfer effectively to others. This research explores unsupervised metrics that assess representation quality based on geometric and algebraic properties, bypassing the need for labels.

Methodology

The study moves beyond information-theoretic measures to evaluate how easily data can be linearly separated in a stable manner. Three primary perspectives are examined:

  1. The Linear Classifier Perspective (Coherence): Utilizing $\mu_0$-incoherence to measure the alignment of an embedding matrix's singular vectors with the standard basis. Lower incoherence indicates a better-distributed representation for linear classification.
  2. The Numerical Linear Algebra Perspective:
    • Stable Rank: An estimation of the "effective" dimensionality of the data, less sensitive to noise than traditional rank.
  3. Condition Number: A proxy for the stability of learned representations, measuring sensitivity to input changes.
  4. The High-dimensional Probability Perspective (SelfCluster): A novel metric introduced by the authors that compares embedding distributions to a theoretical uniform distribution on a high-dimensional sphere, quantifying how efficiently embeddings fill the space.

Experimental Results

The metrics were tested across computer vision (over 1,000 pre-trained models on ImageNet) and graph embeddings (DeepWalk on 10 diverse datasets).

Vision Benchmarks

On the ImageNet training set, NESum and $\alpha$-ReQ demonstrated the strongest Spearman rank correlations ($\rho = 0.40$) with actual test set accuracy. This suggests that these metrics can serve as reliable proxies for performance during the training phase.

Correlation of various unsupervised metrics with ImageNet accuracy on the training set.

Evaluation on the test set revealed that while correlations can shift, certain metrics remain robust. The novel SelfCluster metric, despite its computational complexity, provides a unique perspective on embedding geometry and its relationship to downstream performance.

Comprehensive comparison of unsupervised metrics, including SelfCluster, against ImageNet test set accuracy across a large model suite.

Graph Embeddings and Stability

In graph-based tasks, Coherence emerged as a top performer, showing a near-perfect correlation ($\rho = 0.99$) with downstream classification across multiple datasets. The study also highlighted the stability of RankMe, which can be accurately estimated with few data points, whereas Coherence is more data-dependent and requires larger batches to be reliable.

Conclusion

The findings suggest that while no single metric is a universal solution, unsupervised metrics like NESum, Stable Rank, and Coherence can robustly estimate embedding quality. These metrics are efficient enough to monitor training in real-time, helping to identify issues like "dimensional collapse" without the need for expensive supervised probes. Future work will address the $O(n^2)$ scalability of the SelfCluster metric and explore using these unsupervised signals directly as optimization objectives during training.

Summary updated

Abstract

Unsupervised learning has recently significantly gained in popularity, especially with deep learning-based approaches. Despite numerous successes and approaching supervised-level performance on a variety of academic benchmarks, it is still hard to train and evaluate SSL models in practice due to the unsupervised nature of the problem. Even with networks trained in a supervised fashion, it is often unclear whether they will perform well when transferred to another domain. Past works are generally limited to assessing the amount of information contained in embeddings, which is most relevant for self-supervised learning of deep neural networks. This works chooses to follow a different approach: can we quantify how easy it is to linearly separate the data in a stable way? We survey the literature and uncover three methods that could be potentially used for evaluating quality of representations. We also introduce one novel method based on recent advances in understanding the high-dimensional geometric structure self-supervised learning. We conduct extensive experiments and study the properties of these metrics and ones introduced in the previous work. Our results suggest that while there is no free lunch, there are metrics that can robustly estimate embedding quality in an unsupervised way.
BibTeX
@inproceedings{tsitsulin2023unsupervised,
  author = {Anton Tsitsulin and Marina Munkhoeva and Bryan Perozzi},
  title = {{Unsupervised Embedding Quality Evaluation}},
  booktitle = {TAG-ML},
  year = {2023},
  url = {https://openreview.net/forum?id=6KOxzkLull},
}
Figure