Abstract
Self-supervised speech models (S3Ms) are known to encode rich phonetic information, yet how this information is structured remains underexplored. We conduct a comprehensive study across 96 languages to analyze the underlying structure of S3M representations, with particular attention to phonological vectors. We first show that there exist linear directions within the model's representation space that correspond to phonological features. We further demonstrate that the scale of these phonological vectors correlate to the degree of acoustic realization of their corresponding phonological features in a continuous manner. For example, the difference between [d] and [t] yields a voicing vector: adding this vector to [p] produces [b], while scaling it results in a continuum of voicing. Together, these findings indicate that S3Ms encode speech using phonologically interpretable and compositional vectors, demonstrating phonological vector arithmetic.
Introduction
While self-supervised speech models (S3Ms) like WavLM and HuBERT have revolutionized tasks like speech recognition and synthesis, they are often treated as "black boxes." We know they encode phonetic information, but the internal geometry of that information has remained largely opaque.
Researchers have recently discovered that these models organize speech sounds into a structured, linear geometry. Much like the famous "king - man + woman = queen" analogy in text models, these speech models allow for "phonological arithmetic." For example, adding a "voicing" vector to the representation of a [p] sound successfully transforms it into a [b]. This paper explores whether the internal representation space is structured according to linguistic rules, specifically representing phonological features (like voicing, nasality, or place of articulation) as linear directions that can be manipulated mathematically.
Methodology
The study tested two main hypotheses: Directionality, where phonological features are represented as linear directions in the model's latent space; and Scalability, where the magnitude of a vector along these directions corresponds to the intensity of the acoustic feature.
To test these, the authors defined phonological vectors by taking the difference between the mean representations of sounds with and without a specific feature, using the PanPhon database. To verify their findings, they used a neural vocoder (Vocos) to "invert" the modified representations back into audible speech. By adding a "nasal" vector to a non-nasal consonant and resynthesizing it, they could objectively measure if the resulting audio actually sounded more nasal using acoustic metrics like Center of Gravity (COG) and Formant frequencies.
The research utilized two primary datasets: TIMIT, a classic English dataset with high-quality manual phonetic segmentations, and VoxAngeles, a massive cross-linguistic dataset covering 95 languages. This allowed the researchers to test if the "phonological arithmetic" generalizes to sounds and languages the models never encountered during training.
Key Results and Insights
The findings reveal that S3Ms are not just memorizing sounds; they are learning a compositional "language" of phonology. The fact that $r_{[b]} \simeq r_{[p]} + (r_{[d]} - r_{[t]})$ holds true suggests the models have discovered the underlying building blocks of speech.
Phonological Compositionality and Scalability
Unlike traditional linguistic theory which often treats features as binary (+voice or -voice), S3Ms represent them as a continuum. By scaling the phonological vector ($\lambda$), the researchers could create a "sliding scale" of voicing or nasality. There was a strong monotonic correlation between the scale of the added vector and actual acoustic changes in the resynthesized audio.
As shown in the figure above, increasing the "rounding" vector consistently lowered the first three formants (F1, F2, F3), which is the physical hallmark of lip rounding. Similarly, the "voicing" vector showed strong correlations with acoustic metrics across different phonemes.
Layer-wise Hierarchy and Generalization
The "aha moment" for phonology happens in the middle and late layers of the models. Interestingly, vowels peak in clarity earlier in the network than consonants. This suggests consonants require more temporal context to be fully resolved, which happens in deeper layers.
Furthermore, the models demonstrated impressive Zero-Shot Generalization. Even models trained only on English speech successfully discovered phonological vectors that worked for unseen phones in dozens of other languages. In analogy tests, HuBERT and WavLM achieved success rates of 94% and 92% respectively, significantly outperforming standard spectral baselines like MFCCs (19%).
Conclusion and Future Work
This research proves that S3Ms organize information in a phonologically interpretable and compositional manner. This discovery opens the door for controllable speech synthesis, where specific phonological properties can be edited by simply moving vectors in the latent space.
However, some limitations remain. The vocoder itself might introduce biases or "fill in the blanks" during resynthesis. Future work should investigate if these geometric properties hold across a wider range of model architectures, such as autoregressive models, and explore the causal mechanisms that force models to organize information this way during training.
Summary updated