Accounting for Coarticulation in Continuous Speech Representations

This direction is split into other directions: context-dependent acoustic modeling, co-articulation modeling (phonetic context dependency). Its last version stays readable and citable here.

Version 1 of 1651 papers907 conceptsfrozen 2026-10-04

In traditional linguistics, we often imagine speech as a sequence of discrete "beads on a string"—individual phonemes like /b/, /a/, and /t/ lined up in a row. However, the physical reality of human speech is far messier. As we speak, our articulators (the tongue, lips, and vocal folds) are in constant, fluid motion. They begin moving toward the target for the next sound before the current one is finished, a phenomenon known as coarticulation.

This "phonetic bleeding" means that a single phoneme never sounds the same twice. Its acoustic signature is "colored" by its neighbors. For decades, researchers like Mary O'Kane have noted that this effect extends even across word boundaries in rapid speech, making the discovery of stable, invariant speech units one of the most significant hurdles in the field.

The Failure of Context Invariance

The core challenge for modern AI is achieving context-invariance: the ability to recognize that two vastly different acoustic signals represent the same linguistic unit.

Recent evaluations by Mark Hallap and Ewan Dunbar in Evaluating context-invariance in unsupervised speech representations reveal that even state-of-the-art self-supervised models struggle here. These models experience a three-to-fourfold increase in error rates when identifying phonemes across different phonetic environments compared to identical ones.

This suggests that models often capture "local acoustic snapshots" rather than the abstract phonetic categories humans use. As Abdellah Fourtassi and Emmanuel Dupoux argue in A Rudimentary Lexicon and Semantics Help Bootstrap Phoneme Acquisition, purely bottom-up statistical clustering fails because it cannot distinguish between meaningful phonetic differences (like "pin" vs "bin") and the "noise" of allophonic variation caused by context.

Modeling the Neighborhood

To solve this, researchers have moved toward context-dependent modeling. Instead of looking at a single sound, models look at "tri-phones" or "tri-tones"—units defined by the sound itself plus its immediate left and right neighbors.

In tonal languages like Mandarin, this is critical. Jie Hao demonstrated in A tone recognition framework for continuous Mandarin speech that the pitch contour of a syllable is frequently distorted by its neighbors, with the preceding syllable exerting a stronger "carryover" influence than the succeeding one. Similarly, Mei-Yuh Hwang and Xin Lei found that using a wider window of acoustic context to extract "tone posteriors" significantly reduces errors in conversational speech, where rules like tone sandhi cause tones to shift based on their environment.

This context-dependency isn't limited to pitch. Dong Yu and colleagues showed in Improved Bottleneck Features Using Pretrained Deep Neural Networks that using "senones"—context-dependent states—as training targets allows deep neural networks to capture the fine-grained nuances that monophone models miss.

The Visual Smear

Coarticulation is not just an auditory problem; it is a physical one visible on the face. When we say "pool," our lips begin to round for the "oo" vowel while we are still pronouncing the "p."

In Visual analysis of lip coarticulation in VCV utterances, Aseel Turkmani and Adrian Hilton showed that even during the "closure" phase of a consonant (when the mouth is shut), the lips maintain distinct shapes that reveal the surrounding vowels.

Modern synthesis models, such as the Context-Aware Lip-Sync (CALS) framework by Se Jin Park and Minsu Kim, now use temporal transformers to look at a 1.2-second window of audio. This allows the model to "anticipate" upcoming sounds, creating smoother, more realistic facial animations that avoid the jerky transitions of context-blind models.

Beyond Discrete Units

Perhaps the most radical shift in thinking comes from James R. Glass and Karen Livescu. In Feature-based Pronunciation Modeling for Speech Recognition, they argue that we should stop trying to find discrete phonemes altogether.

Instead, they model speech as a collection of parallel, overlapping streams of articulatory features—like lip opening, tongue height, and glottal state. These features can drift out of sync (asynchrony), naturally explaining why sounds merge or disappear in casual conversation.

This view aligns with recent findings by Dan Wells in Phonetic Analysis of Self-supervised Representations of English Speech, which show that the "units" discovered by models like HuBERT are not phonemes, but sub-phonetic events. These units prioritize the manner of articulation and capture the physical reality of the acoustic signal as it is actually produced in context, rather than trying to force it into an idealized dictionary pronunciation.

Papers in this version

Showing 60 of 651 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_accounting_for_coarticulation_in_continu_f5646cd6,
  title = {Accounting for Coarticulation in Continuous Speech Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/accounting-for-coarticulation-in-continuous-speech-representations/rdr_f5646cd64b3746bc89c27b4f6893d72c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_accounting_for_coarticulation_in_continu_f5646cd6_v1,
  title = {Accounting for Coarticulation in Continuous Speech Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/accounting-for-coarticulation-in-continuous-speech-representations/rdr_f5646cd64b3746bc89c27b4f6893d72c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}