Introduction
Self-supervised learning (SSL) has fundamentally changed speech processing, yet the internal mechanisms of models like HuBERT and wav2vec 2.0 remain largely opaque. Traditional interpretability methods often rely on training "probes"—small downstream classifiers—to extract information from SSL representations. However, this approach can be misleading; a powerful non-linear probe might extract information that is not actually well-organized or easily accessible within the original representation. This research shifts the focus toward the intrinsic spatial organization of these models, investigating how phonetic and speaker information is clustered in the raw embedding space without additional training.
Methodology
To analyze the geometric structure of SSL representations, the researchers employed two primary intrinsic metrics:
- Linear Centered Kernel Alignment (CKA): This metric measures the similarity between the geometric structure of a model layer and specific acoustic features, such as F0 (pitch) or F1/F2 formants. It quantifies how closely the distances between samples in the embedding space mirror distances in the acoustic feature space.
- Multivariate Wilcoxon-Mann-Whitney (WMW) Test (AvgU): This metric quantifies clustering by examining the nearest neighbors of a given sample. If neighbors consistently belong to the same class (e.g., the same phone or speaker), the $AvgU$ score increases, indicating high segregation in the embedding space.
Datasets
The study utilized two distinct speech corpora to ensure findings were robust across different speech qualities:
- TIMIT: A standard dataset of native English speakers with precise, manually-checked phone alignments.
- L2Arctic: A corpus of non-native English speakers, used to evaluate if organizational patterns persist across varied accents and pronunciation qualities.
Key Insights: The Geometry of Speech
The analysis revealed several fundamental properties of how SSL models organize information:
Objective Over Architecture
A striking finding was that a model's training objective is a more significant determinant of its internal organization than its specific architecture. Models sharing the same learning strategy—such as HuBERT and WavLM, which both predict quantized MFCCs—exhibited remarkably similar internal structures, even when trained on different data.
The Speaker-Phone Trade-off
The embedding space faces a natural tension: clustering by speaker tends to disperse instances of the same phone, and vice versa. While models encode both types of information, the spatial organization typically prioritizes one over the other depending on the layer depth. Visualizing these embeddings reveals how distinct clusters form for speaker identity versus phonetic content.
Internal vs. External Targets
Models predicting "internal" targets (e.g., wav2vec 2.0 or data2vec) showed a significant drop in phonetic clustering in their final layers. This suggests that the terminal layers of these models become highly specialized for their specific pretext tasks, potentially at the expense of general acoustic feature representation.
Key Results and the "Middle Layer" Phenomenon
The researchers identified a consistent "sweet spot" for phonetic information in the middle layers (typically layers 2–4 or 6–9) across most models. In these layers, phonetic "purity" is highest before the representations specialize for the pre-training task in the final layers.
The $AvgU$ metric proved to be a powerful predictor of downstream utility. Phonetic clustering scores showed a high correlation (0.84) with performance on the SUPERB phone recognition benchmark. Similarly, speaker clustering correlated at 0.75 with speaker identification tasks. This demonstrates that the intrinsic spatial organization of the raw embedding space is directly linked to a model's practical effectiveness.
Conclusion
By moving beyond probes and examining the raw geometry of SSL representations, this work provides a clearer picture of how speech models "think." The findings highlight the importance of training objectives and the existence of a hierarchical organization where phonetic information peaks in middle layers. Future research may extend these methods to analyze consonants, articulatory features, and the behavior of larger model architectures.
Summary updated