When we read a book or listen to a scripted speech, language flows linearly. But natural human conversation is messy. We pause, repeat words, leave thoughts half-finished, and correct ourselves mid-sentence. Consider an everyday utterance like: "Go to the left—no, sorry, the right corridor."
To a human listener, this correction is seamless. To an Automatic Speech Recognition (ASR) or Natural Language Processing (NLP) system, it is a structural nightmare. Standard models expect a continuous, grammatically coherent sequence of tokens. Spontaneous speech, however, is filled with disfluencies: filled pauses ("um", "uh"), repetitions, stutters, and speech repairs.
Understanding how to model these disfluencies is crucial because they are not random noise; they follow structural, acoustic, and even physical patterns that reveal a speaker's cognitive state and communicative intent.
The Anatomy of a Speech Repair
To process a disfluency, a system must break it down into its constituent parts. Linguists typically divide a speech repair into three distinct stages:
- The Reparandum: The segment of speech that the speaker intends to discard or correct (e.g., "to the left").
- The Interruption Point (IP): The exact moment the speaker stops their forward momentum (often marked by a pause or filler word like "no, sorry").
- The Repair: The corrected phrase that replaces the reparandum (e.g., "the right corridor").
As Shaolei Wang et al. note in Transition-Based Disfluency Detection using LSTMs, identifying the reparandum is exceptionally challenging because it often shares a parallel syntactic structure with the subsequent repair, varies significantly in length, and can be nested within other disfluencies.
Furthermore, speakers frequently use fragmented words—cutting off a word before it is fully pronounced. In Learning to Identify Fragmented Words in Spoken Discourse, Piroska Lendvai demonstrates that these fragments serve as critical early signals that a speaker is about to initiate a self-repair or rephrase their utterance.
Why Disfluencies Break Downstream Systems
When ASR systems encounter these structural disruptions, their performance degrades sharply. This creates a critical bottleneck across various domains:
- Professional and Specialized Domains: In medical environments where precision is vital, the false starts and repetitions common in spontaneous clinician speech increase word error rates and risk losing critical contextual information, as highlighted by Ali Can Kocabiyikoglu et al. in A Spoken Drug Prescription Dataset in French for Spoken Language Understanding. Similarly, unstructured meeting speech with overlapping speakers and fillers complicates transcription and summarization (Using the Amazon Mechanical Turk to Transcribe and Annotate Meeting Speech for Extractive Summarization).
- Educational and Assessment Tools: Automated language learning tools struggle to evaluate non-native speakers because ASR models fail to accurately capture the specific grammatical errors and disfluencies characteristic of second-language learners (The Impact of ASR on the Automatic Analysis of Linguistic Complexity and Sophistication in Spontaneous L2 Speech). In literacy tutoring, reading miscues like repetitions or mispronunciations account for 30% of all speech recognition errors, meaning the technology is least reliable exactly when a child stumbles (Analysis and Detection of Reading Miscues for Interactive Literacy Tutors).
- Clinical and Accessibility Applications: For individuals with speech disabilities or cognitive impairments, standard ASR models frequently omit or completely fail to transcribe disfluent segments like stutters (The Interspeech 2025 Speech Accessibility Project Challenge). Interestingly, Franziska Braun et al. reveal a paradox in Pitfalls and Limits in Automatic Dementia Assessment): large-scale speech models can exhibit higher error rates on rapid, fluent speech due to "repetition hallucinations," whereas the pauses typical of pathological speech can inadvertently improve audio segmentation.
Algorithmic Strategies for Disfluency Detection
To resolve these issues, researchers have developed specialized architectures that move beyond treating speech as a simple linear sequence of text.
Incremental and Syntactic Parsing
Rather than relying on expensive and noisy full syntactic parsing, the authors of Transition-Based Disfluency Detection using LSTMs proposed a transition-based framework using LSTMs. By treating disfluency detection as a series of incremental actions (such as deciding whether to keep or discard a word as it arrives), the model captures chunk-level features and long-range dependencies to isolate the reparandum.
Alternatively, Tim Miller and William Schuler in A Syntactic Time-Series Model for Parsing Fluent and Disfluent Speech introduced a Hierarchical Hidden Markov Model (HHMM). This model utilizes a "right-corner transform" on syntactic trees, allowing the system to parse fluent and disfluent speech simultaneously by explicitly tracking when a syntactic structure has been aborted.
Leveraging Prosodic and Multimodal Cues
Textual transcripts often lack explicit markers for speech repairs. To bridge this gap, researchers look to acoustic and physical signals. In Improved spontaneous Mandarin speech recognition by disfluency interruption point (IP) detection using prosodic features, Che-Kuang Lin and Lin-shan Lee demonstrated that incorporating prosodic interruption probabilities (such as sudden shifts in pitch or duration) into the rescoring of word graphs significantly improves ASR character accuracy.
Taking multimodality a step further, Jacob Eisenstein in Semantic Back-Pointers from Gesture proposed using hand gesture similarity as a "semantic back-pointer." By tracking when a speaker returns their hand to a specific spatial location, a system can visually detect when a speaker has abandoned a thought and reset their communication, providing a physical anchor for speech restarts.
Downstream Correction and Speech Editing
Once disfluencies can be accurately localized, systems can either adapt downstream tasks or actively repair the audio. For example, the structural disruptions caused by spontaneous speech repairs frequently cause errors in breaking conversations into meaningful units, a challenge that requires robust annotation schemes (Intercoder reliability in annotating complex disfluencies) and weakly supervised segmentation models (Weakly supervised discourse segmentation for multiparty oral conversations).
Conversely, instead of just transcribing the messiness, generative models can now clean it up. In FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models, Ziyue Jiang et al. introduced a generative framework that combines a dedicated stutter predictor with a context-aware diffusion model. This architecture localizes complex stuttering patterns and automatically removes them, transforming disfluent recordings into clean, natural-sounding audio without requiring manual waveform editing.
Key Foundational Work
Processing Self Corrections in a speech to speech system (2000) — contributed 1 concept elements to this direction.
Multi-lingual Translation of Spontaneously Spoken Language in a Limited Domain (1996) — contributed 1 concept elements to this direction.