Abstract
This study explores using embedding rank as an unsupervised evaluation metric for general-purpose speech encoders trained via self-supervised learning (SSL). Traditionally, assessing the performance of these encoders is resource-intensive and requires labeled data from the downstream tasks. Inspired by the vision domain, where embedding rank has shown promise for evaluating image encoders without tuning on labeled downstream data, this work examines its applicability in the speech domain, considering the temporal nature of the signals. The findings indicate rank correlates with downstream performance within encoder layers across various downstream tasks and for in- and out-of-domain scenarios. However, rank does not reliably predict the best-performing layer for specific downstream tasks, as lower-ranked layers can outperform higher-ranked ones. Despite this limitation, the results suggest that embedding rank can be a valuable tool for monitoring training progress in SSL speech models, offering a less resource-demanding alternative to traditional evaluation methods.
Introduction
Self-supervised learning (SSL) has revolutionized speech processing by allowing models like HuBERT to learn from massive amounts of unlabeled audio. However, evaluating these models remains a significant bottleneck. Traditionally, researchers must "probe" or fine-tune the model on downstream tasks—such as speech recognition or speaker identification—to assess improvement. This process is computationally expensive, slow, and requires labeled data. Furthermore, the internal loss function used during SSL training doesn't always correlate perfectly with downstream performance. This paper adapts "embedding rank" from computer vision to the temporal nature of speech to create an unsupervised evaluation metric.
Methodology: RankMe-t
The core concept is RankMe, a metric measuring the "effective rank" of the matrix formed by a model's embeddings. A higher rank suggests the model is using its available dimensions effectively, while a low rank indicates "dimensional collapse," where representations are redundant.
Because speech signals are temporal sequences of frames, the authors propose RankMe-t. They extend the original algorithm by extracting embedding sequences from HuBERT layers, applying zero-padding for varying lengths, and summing or mean-pooling the matrices to create a fixed-size representation. The effective rank is then calculated based on the entropy of the singular values of the resulting matrix.
Experimental Setup
Models were pre-trained on the LibriSpeech 960h dataset. Evaluation spanned several downstream tasks:
- Phoneme Recognition (PR): LibriSpeech (in-domain) and TIMIT (out-of-domain).
- Keyword Spotting (KS): Speech Commands dataset.
- Speaker Identification (SID): VoxCeleb1 dataset.
Key Insights and Results
Rank as a Training Monitor
RankMe-t increases steadily as training progresses across all layers. This makes it a "free" diagnostic tool to monitor whether a model is still learning or has plateaued, without requiring labeled datasets.
Within-Layer Correlation and the Layer Paradox
The authors found significant positive correlations (Kendall’s $\tau$ ranging from 0.34 to 0.75) between RankMe-t and downstream performance within individual layers. For any single layer, an increase in rank almost always correlates with improved downstream accuracy.
However, rank is not a global predictor across different layers—a phenomenon termed the "Layer Paradox." A deeper layer (e.g., Layer 9) might have a much higher rank than a shallower one (e.g., Layer 3), yet the shallower layer might perform better on specific tasks. For instance, Layer 6 was found to be optimal for phoneme recognition, while Layer 3 was superior for speaker identification, regardless of their respective ranks. This suggests that while rank tracks improvement within a layer, it does not account for the type of information (e.g., phonetic vs. speaker-specific) that different layers prioritize.
Conclusion
RankMe-t offers a valuable, resource-efficient tool for monitoring the training progress of SSL speech models. While it effectively predicts performance improvements within a specific layer, it cannot yet be used for cross-layer model selection or to determine the "best" layer for a specific application. Future work remains to uncover what information is encoded in these embeddings beyond mathematical rank.
Summary updated