Comparative layer-wise analysis of self-supervised speech models

Ankita Pasad, Bowen Shi, Karen Livescu

Abstract

Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.

Introduction

Self-supervised learning (SSL) has revolutionized speech processing, providing powerful pre-trained models like wav2vec 2.0 and HuBERT that excel at downstream tasks. However, these models often remain "black boxes." This paper provides a much-needed "under the hood" look, revealing how different training objectives change where and how phonetic and semantic information is stored within a model's layers.

Methodology

The core tool used is Projection-Weighted Canonical Correlation Analysis (PWCCA). Unlike "probing," which trains a new classifier on top of a layer, CCA is a lightweight statistical tool that measures the similarity between two sets of representations. The authors compared the representations in each transformer layer against three external variables: acoustic (Mel filter bank features), phonetic (phone labels), and word-level (word identity labels).

By calculating the CCA similarity at every layer, they could visualize the "flow" of information through the network. They then correlated these CCA scores with actual performance on downstream tasks to see if their analysis could predict which layers are most useful.

Information Evolution across Layers

The analysis reveals that a model's pre-training objective dictates where information is concentrated.

For most models, acoustic information is highest in the initial layers and gradually decreases as the representations become more abstract in deeper layers. This trend is consistent across various architectures, as the model moves away from raw input features.

Acoustic similarity (CCA) across layers for various 12-layer models.

The "Autoencoder" vs. "Predictive" Split

Models exhibit distinct patterns for phonetic and word-level information based on their training objectives:

  • Autoencoder-style: Models like wav2vec 2.0 and XLSR-53 (which use a contrastive loss to recover local features) show a trend where phonetic and word information peaks in the middle layers and then drops in the final layers as the model focuses back on the local pretext task.
  • Predictive Models: Models like HuBERT and WavLM (which predict discrete units learned from intermediate layers) concentrate phonetic and word information in their later layers.
Evolution of phonetic/word-level information across layers, illustrating the difference between contrastive and predictive objectives.

Downstream Task Correlation and Efficiency

One of the most significant findings is that using a single intermediate layer often matches or even beats the standard practice of using a weighted sum of all layers. For ASR, using the "best" single layer improved performance by 4-9% relatively over using all layers.

The authors found a high correlation (Spearman’s $\rho \approx 0.90$) between a layer's CCA word-content score and its performance on semantic tasks like intent classification. This means CCA can be used as a "compass" to find the best layers for a task without expensive downstream training. By identifying that the top few layers of models like wav2vec 2.0-large actually lose linguistic information, researchers can discard those layers for certain tasks, saving memory and computation.

Conclusion

This work demonstrates that CCA is a powerful tool for understanding the internal dynamics of speech SSL models. The choice of pre-training objective fundamentally shapes the linguistic hierarchy within the model. These insights not only help in understanding why certain models perform better on specific tasks but also provide a practical guide for more efficient model deployment by selecting optimal layers for downstream applications.

Summary updated

Abstract

Many self-supervised speech models, varying in their pre-training objective, input modality, and pre-training data, have been proposed in the last few years. Despite impressive successes on downstream tasks, we still have a limited understanding of the properties encoded by the models and the differences across models. In this work, we examine the intermediate representations for a variety of recent models. Specifically, we measure acoustic, phonetic, and word-level properties encoded in individual layers, using a lightweight analysis tool based on canonical correlation analysis (CCA). We find that these properties evolve across layers differently depending on the model, and the variations relate to the choice of pre-training objective. We further investigate the utility of our analyses for downstream tasks by comparing the property trends with performance on speech recognition and spoken language understanding tasks. We discover that CCA trends provide reliable guidance to choose layers of interest for downstream tasks and that single-layer performance often matches or improves upon using all layers, suggesting implications for more efficient use of pre-trained models.
BibTeX
@inproceedings{pasad2023comparative,
  author = {Ankita Pasad and Bowen Shi and Karen Livescu},
  title = {{Comparative layer-wise analysis of self-supervised speech models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=vtY6UHwDnC},
}

Loading related content...

Figure