Modeling Disfluencies and Speech Repairs in Spontaneous Speech

39 concepts35 papers2 proposals

When we read a book or listen to a scripted speech, language flows linearly. But natural human conversation is messy. We pause, repeat words, leave thoughts half-finished, and correct ourselves mid-sentence. Consider an everyday utterance like: "Go to the left—no, sorry, the right corridor."

To a human listener, this correction is seamless. To an Automatic Speech Recognition (ASR) or Natural Language Processing (NLP) system, it is a structural nightmare. Standard models expect a continuous, grammatically coherent sequence of tokens. Spontaneous speech, however, is filled with disfluencies: filled pauses ("um", "uh"), repetitions, stutters, and speech repairs.

Understanding how to model these disfluencies is crucial because they are not random noise; they follow structural, acoustic, and even physical patterns that reveal a speaker's cognitive state and communicative intent.

The Anatomy of a Speech Repair

To process a disfluency, a system must break it down into its constituent parts. Linguists typically divide a speech repair into three distinct stages:

  1. The Reparandum: The segment of speech that the speaker intends to discard or correct (e.g., "to the left").
  2. The Interruption Point (IP): The exact moment the speaker stops their forward momentum (often marked by a pause or filler word like "no, sorry").
  3. The Repair: The corrected phrase that replaces the reparandum (e.g., "the right corridor").

As Shaolei Wang et al. note in Transition-Based Disfluency Detection using LSTMs, identifying the reparandum is exceptionally challenging because it often shares a parallel syntactic structure with the subsequent repair, varies significantly in length, and can be nested within other disfluencies.

Furthermore, speakers frequently use fragmented words—cutting off a word before it is fully pronounced. In Learning to Identify Fragmented Words in Spoken Discourse, Piroska Lendvai demonstrates that these fragments serve as critical early signals that a speaker is about to initiate a self-repair or rephrase their utterance.

Why Disfluencies Break Downstream Systems

When ASR systems encounter these structural disruptions, their performance degrades sharply. This creates a critical bottleneck across various domains:

Algorithmic Strategies for Disfluency Detection

To resolve these issues, researchers have developed specialized architectures that move beyond treating speech as a simple linear sequence of text.

Incremental and Syntactic Parsing

Rather than relying on expensive and noisy full syntactic parsing, the authors of Transition-Based Disfluency Detection using LSTMs proposed a transition-based framework using LSTMs. By treating disfluency detection as a series of incremental actions (such as deciding whether to keep or discard a word as it arrives), the model captures chunk-level features and long-range dependencies to isolate the reparandum.

Alternatively, Tim Miller and William Schuler in A Syntactic Time-Series Model for Parsing Fluent and Disfluent Speech introduced a Hierarchical Hidden Markov Model (HHMM). This model utilizes a "right-corner transform" on syntactic trees, allowing the system to parse fluent and disfluent speech simultaneously by explicitly tracking when a syntactic structure has been aborted.

Leveraging Prosodic and Multimodal Cues

Textual transcripts often lack explicit markers for speech repairs. To bridge this gap, researchers look to acoustic and physical signals. In Improved spontaneous Mandarin speech recognition by disfluency interruption point (IP) detection using prosodic features, Che-Kuang Lin and Lin-shan Lee demonstrated that incorporating prosodic interruption probabilities (such as sudden shifts in pitch or duration) into the rescoring of word graphs significantly improves ASR character accuracy.

Taking multimodality a step further, Jacob Eisenstein in Semantic Back-Pointers from Gesture proposed using hand gesture similarity as a "semantic back-pointer." By tracking when a speaker returns their hand to a specific spatial location, a system can visually detect when a speaker has abandoned a thought and reset their communication, providing a physical anchor for speech restarts.

Downstream Correction and Speech Editing

Once disfluencies can be accurately localized, systems can either adapt downstream tasks or actively repair the audio. For example, the structural disruptions caused by spontaneous speech repairs frequently cause errors in breaking conversations into meaningful units, a challenge that requires robust annotation schemes (Intercoder reliability in annotating complex disfluencies) and weakly supervised segmentation models (Weakly supervised discourse segmentation for multiparty oral conversations).

Conversely, instead of just transcribing the messiness, generative models can now clean it up. In FluentSpeech: Stutter-Oriented Automatic Speech Editing with Context-Aware Diffusion Models, Ziyue Jiang et al. introduced a generative framework that combines a dedicated stutter predictor with a context-aware diffusion model. This architecture localizes complex stuttering patterns and automatically removes them, transforming disfluent recordings into clean, natural-sounding audio without requiring manual waveform editing.

Key Foundational Work

Processing Self Corrections in a speech to speech system (2000) — contributed 1 concept elements to this direction.
Multi-lingual Translation of Spontaneously Spoken Language in a Limited Domain (1996) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_disfluencies_in_spontaneous_speech_f4fa9270,
  title = {disfluencies in spontaneous speech},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disfluencies-in-spontaneous-speech/rdr_f4fa9270bf6c45598dbc44ccededb511},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_disfluencies_in_spontaneous_speech_f4fa9270_v1,
  title = {disfluencies in spontaneous speech},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/disfluencies-in-spontaneous-speech/rdr_f4fa9270bf6c45598dbc44ccededb511/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake