Phone and speaker spatial organization in self-supervised speech representations

Pablo Riera, Manuela Cerdeiro, Leonardo Pepino, Luciana Ferrer

Introduction

Self-supervised learning (SSL) has fundamentally changed speech processing, yet the internal mechanisms of models like HuBERT and wav2vec 2.0 remain largely opaque. Traditional interpretability methods often rely on training "probes"—small downstream classifiers—to extract information from SSL representations. However, this approach can be misleading; a powerful non-linear probe might extract information that is not actually well-organized or easily accessible within the original representation. This research shifts the focus toward the intrinsic spatial organization of these models, investigating how phonetic and speaker information is clustered in the raw embedding space without additional training.

Methodology

To analyze the geometric structure of SSL representations, the researchers employed two primary intrinsic metrics:

  1. Linear Centered Kernel Alignment (CKA): This metric measures the similarity between the geometric structure of a model layer and specific acoustic features, such as F0 (pitch) or F1/F2 formants. It quantifies how closely the distances between samples in the embedding space mirror distances in the acoustic feature space.
  2. Multivariate Wilcoxon-Mann-Whitney (WMW) Test (AvgU): This metric quantifies clustering by examining the nearest neighbors of a given sample. If neighbors consistently belong to the same class (e.g., the same phone or speaker), the $AvgU$ score increases, indicating high segregation in the embedding space.

Datasets

The study utilized two distinct speech corpora to ensure findings were robust across different speech qualities:

  • TIMIT: A standard dataset of native English speakers with precise, manually-checked phone alignments.
  • L2Arctic: A corpus of non-native English speakers, used to evaluate if organizational patterns persist across varied accents and pronunciation qualities.

Key Insights: The Geometry of Speech

The analysis revealed several fundamental properties of how SSL models organize information:

Objective Over Architecture

A striking finding was that a model's training objective is a more significant determinant of its internal organization than its specific architecture. Models sharing the same learning strategy—such as HuBERT and WavLM, which both predict quantized MFCCs—exhibited remarkably similar internal structures, even when trained on different data.

The Speaker-Phone Trade-off

The embedding space faces a natural tension: clustering by speaker tends to disperse instances of the same phone, and vice versa. While models encode both types of information, the spatial organization typically prioritizes one over the other depending on the layer depth. Visualizing these embeddings reveals how distinct clusters form for speaker identity versus phonetic content.

Visualizing the spatial organization of embeddings across datasets, sex, speaker IDs, and phones.

Internal vs. External Targets

Models predicting "internal" targets (e.g., wav2vec 2.0 or data2vec) showed a significant drop in phonetic clustering in their final layers. This suggests that the terminal layers of these models become highly specialized for their specific pretext tasks, potentially at the expense of general acoustic feature representation.

Key Results and the "Middle Layer" Phenomenon

The researchers identified a consistent "sweet spot" for phonetic information in the middle layers (typically layers 2–4 or 6–9) across most models. In these layers, phonetic "purity" is highest before the representations specialize for the pre-training task in the final layers.

Average clustering (AvgU) for speakers and phones across layers for various SSL models.

The $AvgU$ metric proved to be a powerful predictor of downstream utility. Phonetic clustering scores showed a high correlation (0.84) with performance on the SUPERB phone recognition benchmark. Similarly, speaker clustering correlated at 0.75 with speaker identification tasks. This demonstrates that the intrinsic spatial organization of the raw embedding space is directly linked to a model's practical effectiveness.

Conclusion

By moving beyond probes and examining the raw geometry of SSL representations, this work provides a clearer picture of how speech models "think." The findings highlight the importance of training objectives and the existence of a hierarchical organization where phonetic information peaks in middle layers. Future research may extend these methods to analyze consonants, articulatory features, and the behavior of larger model architectures.

Summary updated

Abstract

Self-supervised representations of speech are currently being widely used for a large number of applications. Recently, some efforts have been made in trying to analyze the type of information present in each of these representations. Most such work uses downstream models to test whether the representations can be successfully used for a specific task. The downstream models, though, typically perform nonlinear operations on the representation extracting information that may not have been readily available in the original representation. In this work, we analyze the spatial organization of phone and speaker information in several state-of-the-art speech representations using methods that do not require a downstream model. We measure how different layers encode basic acoustic parameters such as formants and pitch using representation similarity analysis. Further, we study the extent to which each representation clusters the speech samples by phone or speaker classes using non-parametric statistical testing. Our results indicate that models represent these speech attributes differently depending on the target task used during pretraining.
BibTeX
@inproceedings{riera2023phone,
  author = {Pablo Riera and Manuela Cerdeiro and Leonardo Pepino and Luciana Ferrer},
  title = {{Phone and speaker spatial organization in self-supervised speech representations}},
  booktitle = {Workshops @ ICASSP},
  year = {2023},
  doi = {10.1109/icasspw59220.2023.10193460},
}
Figure