Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

Kwanghee Choi, Eunjung Yeo, Cheol Jun Cho, David R. Mortensen, David Harwath

Abstract

Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.

Introduction

Transformer-based self-supervised speech models (S3Ms) like wav2vec 2.0, HuBERT, and WavLM have revolutionized speech processing. While these models are known to be "contextualized"—meaning they leverage surrounding acoustic information to interpret the current frame—the internal organization of this context within their high-dimensional vector spaces has remained a "black box." Previous research established that S3Ms represent individual phones compositionally (e.g., the vector for [m] is a sum of "voiced," "bilabial," and "nasal" features). This paper investigates how models maintain this compositionality across sequences of phones without information interference.

Methodology

The study analyzed three major S3Ms (wav2vec 2.0, HuBERT, and WavLM) using the TIMIT and VoxAngeles datasets. The researchers employed several techniques to probe the models' internal representations:

  1. Phonological Analogies: Testing if linguistic features are linear via vector arithmetic (e.g., [d] - [t] + [p] ≈ [b]).
  2. Contextual Probing: Extending these analogies to neighboring phones to see if a current frame encodes features of the previous or next phone.
  3. Subspace Analysis: Measuring the cosine similarity between phonological vectors extracted for different relative positions (-1, 0, +1).
  4. Boundary Tracking: Analyzing representation shifts at phonetic boundaries to determine if the model's context window respects linguistic segments.
Conceptual illustration of the hypothesis and empirical visualization of orthogonal subspaces.

Key Insights

The Orthogonal Subspace Solution

A central finding is that S3Ms utilize distinct, orthogonal subspaces for different relative positions. The vector representing a feature (like "voicing") for the current phone is nearly perpendicular to the vector for the same feature in the next phone. This allows the model to superimpose information from multiple phones into a single 20ms frame without cross-talk.

Implicit Segmentation and the "Staircase Effect"

The models do not use a simple fixed-time sliding window. Instead, they exhibit sensitivity to phonetic boundaries. Visualizing cosine similarity over time reveals a "staircase" pattern: the representation of the "next phone" remains stable within a segment and shifts precisely at the phonetic boundary, as seen in the heatmaps above.

Key Results

Analogy tests showed high success rates for the current phone (Position 0) and significant encoding of immediate neighbors (±1). WavLM demonstrated the most robust context window, maintaining high success rates for neighbors across several layers.

Success rates of phonological analogies across normalized relative phone positions, showing how context information shifts at boundaries.

As shown in the figure above, WavLM (dark blue) maintains high success rates across a wider range of relative positions compared to other models, indicating a broader and more stable contextual representation that respects phonetic boundaries.

Conclusion

This research demonstrates that S3Ms encode phonetic context through a highly organized, compositional structure of orthogonal subspaces. These findings suggest future applications in unsupervised speech segmentation, more controllable text-to-speech synthesis, and the development of more efficient, disentangled speech architectures. While the storage method is now clearer, future work is needed to understand the specific attention mechanisms that construct these subspaces during training.

Summary updated

Abstract

Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations.
BibTeX
@misc{choi2026self,
  author = {Kwanghee Choi and Eunjung Yeo and Cheol Jun Cho and David R. Mortensen and David Harwath},
  title = {{Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces}},
  year = {2026},
  eprint = {2603.12642},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.12642},
}
Figure