Similarity Analysis of Self-Supervised Speech Representations

Yu-An Chung, Yonatan Belinkov, James R. Glass

Abstract

Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models' pre-training loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.

Introduction

In recent years, the speech community has moved away from hand-crafted features, such as Mel spectrograms, toward learned representations. Algorithms like Contrastive Predictive Coding (CPC), Autoregressive Predictive Coding (APC), and Masked Predictive Coding (MPC) have all shown success, but they are often treated as "black boxes." This research investigates the internal properties of these models to determine if models trained on the same data with different objectives "see" the speech signal the same way, and whether architectural choices like RNNs versus Transformers fundamentally change the resulting representations.

Methodology

The study employs two primary analytical tools to evaluate representations pre-trained on LibriSpeech (1,000 hours) and Libri-Light (up to 6,000 hours):

  1. Similarity Measures: Linear Centered Kernel Alignment (lincka) and Singular Vector Canonical Correlation Analysis (svcca) are used to compare high-dimensional vectors across models with different internal dimensions.
  2. Probing Tasks: Frozen pre-trained models are evaluated using simple linear classifiers for phonetic classification (identifying speech sounds) and speaker classification (identifying the speaker) using the Wall Street Journal (WSJ) and TIMIT datasets.

Analysis of Representation Similarity

A key finding of this research is that the training objective (e.g., future prediction vs. masked prediction) has a significantly higher impact on representation similarity than the specific neural building blocks used. Models within the same "family" (sharing an objective) exhibit high similarity scores, often exceeding 0.85, regardless of whether they utilize RNN or Transformer architectures.

Figure 1: Heatmap showing high similarity within model families (e.g., APC variants) and lower similarity across different objectives.

Directionality also plays a crucial role; whether a model is unidirectional or bidirectional changes the representation more than the choice of building block. Furthermore, scaling experiments suggest that CPC is more efficient at utilizing massive amounts of unlabeled data compared to generative models like APC and MPC, as its representations change more significantly when moving from 2k to 6k hours of training audio.

Correlation Between Pre-training and Downstream Performance

The study investigates whether pre-training loss can serve as a reliable proxy for downstream task performance. For generative models like APC and MPC, there is a near-perfect correlation between pre-training loss and phonetic classification accuracy. This suggests that researchers can use training loss to select the best model checkpoint without performing expensive downstream evaluations.

Figure 4: Pearson correlation coefficients between pre-training loss and probing task performance, showing APC's high correlation with phonetic information.

As shown in the data, APC-fw-rnn achieves a Pearson correlation of 0.989 for phonetic tasks. In contrast, CPC's correlation is lower (0.64 to 0.67), indicating that its contrastive loss is a noisier indicator of the phonetic quality of the learned representations.

Conclusion

This similarity analysis reveals that the "logic" of self-supervised speech models is primarily driven by their training objectives and directionality rather than their specific architectural components. While APC and MPC offer highly predictable performance based on training loss, CPC demonstrates superior scaling capabilities. Future work should expand this analysis to state-of-the-art models like HuBERT and Wav2Vec 2.0 and explore more complex downstream tasks beyond linear probing.

Summary updated

Abstract

Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models' pre-training loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.
BibTeX
@inproceedings{chung2021similarity,
  author = {Yu-An Chung and Yonatan Belinkov and James R. Glass},
  title = {{Similarity Analysis of Self-Supervised Speech Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414321},
}
Figure