Subword Modeling for Disfluent and Disordered Speech

17 concepts10 papers1 proposal

Traditional Automatic Speech Recognition (ASR) systems operate under a fundamental assumption: speech is a sequence of discrete, well-formed words. While this abstraction works well for typical speech, it breaks down entirely when encountering disfluencies or speech disorders. When a speaker stutters ("c-c-cat"), hesitates, or exhibits phonetic distortions due to conditions like dysarthria or aphasia, the breakdown occurs within the word, not between words.

Word-level models act as a lossy compression filter for these variations. Modern end-to-end models often implicitly "correct" or ignore these irregularities to output clean text. However, as Jiachen Lian et al. demonstrate in Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection, whole-word analysis completely overlooks syllable-level repetitions and fine-grained phonetic distortions. To capture the clinical and acoustic reality of atypical speech, models must shift their canvas from the word level to the subword and phoneme level.

Why Phoneme-Level Modeling Works

The core intuition behind subword speech modeling is alignment resolution. A phoneme is the smallest distinct unit of sound in a language. By treating phonemes—rather than words—as the primary modeling targets, speech systems can map acoustic events to their exact temporal boundaries.

Consider a part-word repetition like "st-st-stop". A word-level model struggles to align the prolonged acoustic frames with a single "stop" token. In contrast, a phoneme-level model can explicitly represent the sequence of sounds. As Theodoros Kouzelis et al. show in Weakly-supervised forced alignment of disfluent speech using phoneme-level modeling, modeling speech at the phoneme level eliminates the need for complex syllable segmentation while inherently providing robustness against part-word repetitions and stuttering patterns. This granular representation allows forced aligners to remain stable even when the incoming audio deviates drastically from standard pronunciation templates.

Capturing the Nuances of Hesitation and Pathological Speech

Operating at the subword level allows models to build disfluency-aware grammars. For instance, in child literacy assessment, hesitations often manifest as partial words or isolated phonetic attempts. In Estimation of children's reading ability by fusion of automatic pronunciation verification and fluency detection, Matthew Black et al. utilize phonetic grammars to capture these fragmented sounds, mapping them directly to quantitative fluency metrics.

This approach is equally vital for modeling pathological speech. In conditions like dysarthria, speakers may struggle with specific articulatory movements, leading to distortions of initial consonants or vowels. Phil Green et al. demonstrated in Revisiting dysarthria assessment intelligibility metrics that analyzing speech via pseudo-phones or syllable-initial segments captures the precise decoding effort required by a listener, outperforming whole-word models. By focusing on fine-grained phonetic units, the system preserves the non-speech sounds and variations critical for clinical diagnosis.

Granular Metrics for Intelligibility and Naturalness

Beyond detection and alignment, subword modeling transforms how we measure speech intelligibility and naturalness. Traditional metrics rely on Word Error Rate (WER), which penalizes a partially mispronounced word as a total failure. Subword analysis fixes this by evaluating speech at the phoneme or grapheme level.

In Towards a Comprehensive Assessment of Speech Intelligibility for Pathological Speech, W. Xue et al. show that calculating subword-level accuracy and edit distances provides highly reliable and valid predictions of speech intelligibility. This granularity ensures that if a speaker correctly articulates most phonemes within a word, the model recognizes the partial success. As Mario Ganzeboom et al. note in Intelligibility of Disordered Speech: Global and Detailed Scores, this allows for the identification of correctly perceived speech units even when entire words are unintelligible.

This principle extends directly to non-native speech evaluation. Yasuo Suzuki et al. established in Analysis of the phone level contributions to objective evaluation of English speech by non-natives that phone- and syllable-level timing analysis captures the subtle temporal deviations that dictate how natural a non-native speaker sounds. Similarly, Xing Wei et al. in Assessing Intelligibility in Non-native Speech: Comparing Measures Obtained at Different Levels confirm that converting text to phoneme strings to compute fine-grained distance metrics pinpoints specific pronunciation challenges that word-level metrics completely miss.

Scaling to Modern Language Models

The ultimate goal of this research direction is to integrate these granular insights into large-scale speech models. Early foundational work, such as the vertical alignment scheme proposed by Peter A. Heeman et al. in An annotation scheme for complex disfluencies, laid the groundwork by showing that detailed annotations allow language models to learn intricate disfluency patterns.

Today, this has evolved into end-to-end frameworks. In SSDM: Scalable Speech Dysfluency Modeling, Jiachen Lian et al. demonstrate that moving toward phoneme-level language modeling enables systems to accurately identify, explain, and provide diagnostic feedback on speech impediments within individual sounds. By breaking free from the constraints of word-level timestamps, modern speech AI can finally perceive and reason about speech exactly as it is spoken—imperfections and all.

Key Foundational Work

Revisiting dysarthria assessment intelligibility metrics (2004) — contributed 2 concept elements to this direction.
An annotation scheme for complex disfluencies (2006) — contributed 1 concept elements to this direction.
Analysis of the phone level contributions to objective evaluation of English speech by non-natives (2004) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_phoneme_level_disfluency_modeling_56e8967e,
  title = {phoneme-level disfluency modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phoneme-level-disfluency-modeling/rdr_56e8967ece3e48bea7230148eb5b5f12},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_phoneme_level_disfluency_modeling_56e8967e_v1,
  title = {phoneme-level disfluency modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/phoneme-level-disfluency-modeling/rdr_56e8967ece3e48bea7230148eb5b5f12/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake