Similarity Analysis of Self-Supervised Speech Representations

Yu-An Chung, Yonatan Belinkov, James R. Glass

Self-supervised learning (SSL) has revolutionized speech processing, allowing models to learn powerful representations from thousands of hours of unlabeled audio. However, while we have a growing zoo of algorithms like CPC, APC, and MPC, we still don't fully understand what these models are actually learning or how they relate to one another. This paper provides a much-needed "under the hood" look at these representations.

TL;DR

This paper investigates the internal properties of self-supervised speech models by quantifying how similar their learned representations are and how well their training loss predicts downstream performance. The authors find that a model's training objective (what it is trying to predict) influences its internal "logic" far more than its architecture (whether it uses an RNN or a Transformer).

What Problem Does This Paper Address?

In recent years, the speech community has moved away from hand-crafted features (like Mel spectrograms) toward learned representations. Algorithms like Contrastive Predictive Coding (CPC), Autoregressive Predictive Coding (APC), and Masked Predictive Coding (MPC) have all shown success, but they are often treated as black boxes.

Researchers face several unanswered questions:

  1. If two models are trained on the same data but with different objectives, do they "see" the speech signal the same way?
  2. Does the choice of building block (RNN vs. Transformer) fundamentally change the representation?
  3. Can we use the pre-training loss as a reliable proxy for how well a model will perform on a real task like phoneme recognition?

Methods

The authors employ two primary analytical tools:

  1. Similarity Measures: They use Linear Centered Kernel Alignment (lincka) and Singular Vector Canonical Correlation Analysis (svcca). These are mathematical frameworks that allow researchers to compare two high-dimensional representations (vectors) to see how correlated they are, even if they have different internal dimensions.
  2. Probing Tasks: To see what information is actually "inside" the representations, they freeze the pre-trained models and train simple linear classifiers on top of them to perform:
  3. Phonetic Classification: Identifying the specific speech sound (phone) in each frame.
  4. Speaker Classification: Identifying who is speaking.

The study compares several model variants, including unidirectional vs. bidirectional architectures and different types of neural building blocks (RNNs, Transformers, and CNNs).

Datasets

  • Pre-training: The models were primarily trained on LibriSpeech (1,000 hours of clean English speech). For scaling experiments, they used the Libri-Light (unlab-6k) subset, providing up to 6,000 hours of audio.
  • Probing/Evaluation: Similarity and probing tasks were conducted using the Wall Street Journal (WSJ) and TIMIT datasets, which provide high-quality phonetic and speaker annotations.

Key Insights and Ideas

  • Objective Trumps Architecture: The most striking finding is that the training objective (e.g., predicting the future vs. predicting masked parts) has a much higher impact on representation similarity than whether the model uses an RNN or a Transformer. Models with the same objective look very similar internally, regardless of their "plumbing."
  • Directionality Matters: Whether a model is unidirectional (looking only at the past) or bidirectional (looking at past and future) significantly changes the representation. In fact, directionality affects similarity more than the specific neural architecture used.
  • The "Data Sponge" Effect: While all models improve with more data, CPC (Contrastive Predictive Coding) is found to be more efficient at utilizing massive amounts of unlabeled data compared to generative models like APC and MPC.
  • Loss as a Performance Proxy: For APC and MPC, there is a very strong correlation between the pre-training loss and downstream performance. This means researchers can use the training loss to decide which model "checkpoint" is best without needing to run expensive downstream evaluations every time.

Key Results

  • Similarity Heatmaps: The authors produced heatmaps showing that models within the same "family" (e.g., all APC variants) have high similarity scores (often > 0.85), while similarity across families (e.g., APC vs. CPC) is much lower.
  • Correlation Coefficients: APC showed a near-perfect Pearson correlation (0.989) between its pre-training loss and phonetic classification error. CPC's correlation was lower (around 0.64 to 0.67), suggesting its loss is a noisier indicator of phonetic quality.
  • Scaling: When increasing training data from 2k to 6k hours, CPC representations changed more significantly than APC/MPC, suggesting CPC is better at extracting new, complex features from larger datasets.

Limitations and Future Work

The authors acknowledge that while similarity analysis is powerful, it doesn't provide "absolute facts"—it only tells us how models relate to each other.

Future work could include:

  • Expanding the analysis to newer, state-of-the-art models like HuBERT or Wav2Vec 2.0.
  • Exploring more complex downstream tasks beyond simple linear probing (e.g., full speech recognition or emotion detection).
  • Investigating why CPC scales more efficiently and how to bring that efficiency to generative models like APC.

Summary updated

Abstract

Self-supervised speech representation learning has recently been a prosperous research topic. Many algorithms have been proposed for learning useful representations from large-scale unlabeled data, and their applications to a wide range of speech tasks have also been investigated. However, there has been little research focusing on understanding the properties of existing approaches. In this work, we aim to provide a comparative study of some of the most representative self-supervised algorithms. Specifically, we quantify the similarities between different self-supervised representations using existing similarity measures. We also design probing tasks to study the correlation between the models’ pre-training loss and the amount of specific speech information contained in their learned representations. In addition to showing how various self-supervised models behave differently given the same input, our study also finds that the training objective has a higher impact on representation similarity than architectural choices such as building blocks (RNN/Transformer/CNN) and directionality (uni/bidirectional). Our results also suggest that there exists a strong correlation between pre-training loss and downstream performance for some self-supervised algorithms.
BibTeX
@inproceedings{chung2021similarity,
  author = {Yu-An Chung and Yonatan Belinkov and James R. Glass},
  title = {{Similarity Analysis of Self-Supervised Speech Representations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414321},
}

Loading related content...

Figure