Comparative Layer-Wise Analysis of Self-Supervised Speech Models

Ankita Pasad, Bowen Shi, Karen Livescu

Self-supervised learning (SSL) has revolutionized speech processing, providing powerful pre-trained models like wav2vec 2.0 and HuBERT that excel at downstream tasks. However, these models often remain "black boxes." This paper provides a much-needed "under the hood" look, revealing how different training objectives change where and how phonetic and semantic information is stored within a model's layers.

TL;DR

Researchers used Canonical Correlation Analysis (CCA) to map how acoustic, phonetic, and word-level information evolves across the layers of 11 different self-supervised speech models. They discovered that a model's pre-training objective dictates where information is concentrated and that using a single, well-chosen intermediate layer often outperforms using the entire model for downstream tasks.

What Problem Does This Paper Address?

While SSL models for speech have achieved impressive results, we have a limited understanding of their internal representations. Most practitioners either fine-tune the whole model or use a weighted sum of all layers, often without knowing which layers actually contain the most relevant information for their specific task (e.g., speech recognition vs. intent classification).

The authors address this gap by investigating how different architectures and pre-training tasks—such as contrastive learning (wav2vec 2.0) versus masked prediction of discrete units (HuBERT)—affect the "linguistic hierarchy" within the model.

Methods

The core tool used is Projection-Weighted Canonical Correlation Analysis (PWCCA). Unlike "probing," which trains a new classifier on top of a layer, CCA is a lightweight statistical tool that measures the similarity between two sets of representations.

The authors compared the representations in each transformer layer against three external variables:

  1. Acoustic: Mel filter bank features.
  2. Phonetic: Phone labels.
  3. Word-level: Word identity labels.

By calculating the CCA similarity at every layer, they could visualize the "flow" of information through the network. They then correlated these CCA scores with actual performance on downstream tasks to see if their analysis could predict which layers are most useful.

Datasets

The study analyzes 11 models pre-trained on massive datasets, including:

  • LibriSpeech (960h) & LibriLight (60kh): Standard English speech corpora.
  • GigaSpeech & VoxPopuli: Large-scale diverse speech data.
  • LRS3: Audio-visual speech data.
  • SpokenCOCO: Images paired with spoken descriptions.

For evaluation, they used the SUPERB benchmark (Phonetic Recognition, ASR, Intent Classification) and the SLURP dataset (Spoken Language Understanding).

Key Insights and Ideas

  • The "Autoencoder" vs. "Predictive" Split: Models like wav2vec 2.0 and XLSR-53 (which use a contrastive loss to recover local features) show an "autoencoder-style" trend. Phonetic and word information peaks in the middle layers and then drops in the final layers as the model focuses back on the local pretext task.
  • Predictive Models Push Info Higher: Models like HuBERT and WavLM (which predict discrete units learned from intermediate layers) concentrate phonetic and word information in their later layers.
  • The Single-Layer Advantage: One of the most surprising findings is that using a single intermediate layer often matches or even beats the standard practice of using a weighted sum of all layers. For ASR, using the "best" single layer improved performance by 4-9% relatively over using all layers.
  • CCA as a Guide: The authors found a high correlation (Spearman’s $\rho \approx 0.90$) between a layer's CCA word-content score and its performance on semantic tasks like intent classification. This means CCA can be used as a "compass" to find the best layers for a task without expensive downstream training.

Key Results

  • Phonetic/Word Correlation: CCA-phone and CCA-word scores are highly predictive of performance in Phonetic Recognition (PR) and Automatic Speech Recognition (ASR).
  • Semantic Tasks: For Spoken Language Understanding (SLU), CCA-word scores are much more predictive than CCA-phone scores, confirming that these tasks rely on higher-level abstractions.
  • Efficiency: By identifying that the top few layers of models like wav2vec 2.0-large actually lose linguistic information, researchers can discard those layers for certain tasks, saving memory and computation.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Task Variety: The study focuses on acoustic, phonetic, and word-level properties. Future work could extend this to speaker identity, prosody, or syntax.
  • Language Scope: Most of the analysis is centered on English-centric models; applying this to more multilingual or low-resource models would be valuable.
  • Optimization Issues: They noted that the very last layers of some large models (like wav2vec 2.0-large) failed to converge on ASR tasks despite having decent CCA scores, suggesting that "information content" and "trainability" are not always the same thing.

Summary updated

Abstract

Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models. 1
BibTeX
@inproceedings{pasad2023comparative,
  author = {Ankita Pasad and Bowen Shi and Karen Livescu},
  title = {{Comparative Layer-Wise Analysis of Self-Supervised Speech Models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=vtY6UHwDnC},
}

Loading related content...

Figure