When humans speak spontaneously, our language is rarely clean. We hesitate, repeat words, correct ourselves mid-sentence, and scatter non-lexical fillers like "uh" and "um" throughout our conversations. For decades, the standard paradigm in natural language processing (NLP) and automated speech recognition (ASR) treated these disfluencies as literal noise—structural anomalies to be scrubbed away during preprocessing to yield a clean, grammatically correct transcript.
However, a growing body of research demonstrates that removing disfluencies causes a severe pragmatic and diagnostic signal loss. Far from being random errors, disfluencies are highly structured behavioral cues that map directly to cognitive load, communicative intent, and neurological health. When models filter them out, they lose the very context needed to truly understand human interaction.
The Pragmatic Value of Hesitation
To understand why disfluencies are valuable, we must look at why they occur. In a seminal perspective, Susan Brennan argued in Processes that Shape Conversation and their Implications for Computational Linguistics that spontaneous conversation is an inherently collaborative coordination problem. Disfluencies serve as critical signals that help human listeners anticipate upcoming linguistic challenges and coordinate understanding in real time.
Consider how this manifests in downstream tasks like text summarization. Intuitively, a speaker pauses or uses a filler word when their cognitive load increases—often right before they introduce a highly specific, low-frequency, or high-information concept. By analyzing spontaneous dialogue, Xiaodan Zhu et al. demonstrated in Summarization of spontaneous conversations and Comparing the roles of textual, acoustic and spoken-language features on spontaneous-conversation summarization that features like "uh," "um," and word repetitions are powerful pragmatic markers. Instead of obscuring the core message, these markers actually help models identify and extract high-information content.
Standard Pipeline: "We should meet at... [scrubbed] ...three o'clock." -> (Loses urgency/hesitation)
Disfluency-Aware: "We should meet at... uh... three o'clock." -> (Signals cognitive weight)
Beyond identifying key information, fillers act as discriminative features for paralinguistic and affective states. Tanvi Dinkar et al. showed in The importance of fillers for text representations of speech transcripts that removing filler words significantly degrades a model's ability to detect a speaker's confidence and sentiment. Similarly, Morteza Rohanian et al. found in Disfluent Cues for Enhanced Speech Understanding in Large Language Models that preserving speech repairs provides explicit cues that clarify speaker intent, directly boosting the performance of Large Language Models.
When we strip these elements to maximize text readability, we also erase the social dynamics of conversation. As Hua Shen 0005 et al. noted in MultiTurnCleanup: A Benchmark for Multi-Turn Spoken Conversational Transcript Cleanup, conversational fillers and backchannel responses (like "mhm" or "yeah") are essential indicators of active listening and engagement. Erasing them creates a fundamental trade-off between a clean transcript and an accurate representation of interpersonal meaning. This omission even skews fundamental linguistic modeling; Yu Wang et al. observed in How Much Does Nonverbal Communication Conform to Entropy Rate Constancy?: A Case Study on Listener Gaze in Interaction that omitting disfluencies from information-theoretic analyses of dialogue results in an incomplete understanding of how cognitive load is managed in real-time interaction.
Disfluencies as Clinical Biomarkers
The case for retaining disfluencies becomes even more critical in clinical contexts. Neurological conditions, language disorders, and cognitive decline fundamentally alter the way speech is planned and executed. In these scenarios, the frequency, duration, and placement of pauses and repetitions cease to be mere conversational quirks—they become vital behavioral biomarkers.
In automated dementia and Alzheimer's detection, the preservation of non-lexical tokens is the linchpin of model accuracy. Shahla Farzana et al. explicitly quantified this in How You Say It Matters: Measuring the Impact of Verbal Disfluency Tags on Automated Dementia Detection, proving that removing repetitions and false starts severely degrades the performance of diagnostic models. This occurs because disfluencies capture the exact word-retrieval struggles and cognitive hesitations characteristic of early-stage impairment.
To leverage this signal, researchers have shifted toward disfluency-inclusive architectures:
- Narrative Coherence: Ji Zhu et al. developed an attention-based hybrid network in An Attention-Based Hybrid Network for Automatic Detection of Alzheimer's Disease from Narrative Speech that processes transcripts without removing fillers or pauses, allowing the model to capture subtle patterns of narrative degradation.
- Multi-Class Screening: Nadine Abdelhalim et al. demonstrated in Training Models on Oversampled Data and a Novel Multi-class Annotation Scheme for Dementia Detection that explicitly retaining paralinguistic markers allows models to successfully differentiate between healthy speech, Mild Cognitive Impairment (MCI), and advanced dementia.
- Multimodal Alignment: In language disorders like aphasia, Daeun Lee et al. showed in Learning Co-Speech Gesture for Multimodal Aphasia Type Detection that filtering out disfluencies eliminates the exact moments where speech and hand gestures correlate most strongly, stripping the model of its most informative diagnostic data.
The Upstream Bottleneck of Automated Speech Recognition
If disfluencies are so valuable, why are they routinely discarded? The root of the problem lies in an upstream optimization mismatch within standard ASR systems.
Traditional ASR models are trained to minimize Word Error Rate (WER) against polished, fluent target text. Consequently, they are designed to automatically "clean up" speech by omitting word fragments, hesitations, and repetitions. As Dominik Wagner et al. pointed out in Large Language Models for Dysfluency Detection in Stuttered Speech, this creates a paradox: highly optimized transcription systems mask the very diagnostic evidence required to detect and analyze speech disorders like stuttering.
Acoustic Signal: "I want to go to the th- th- the park."
Standard ASR: "I want to go to the park." (Clean, but masks stutter)
Disfluency ASR: "I want to go to the th- th- the park." (Preserves diagnostic data)
This loss of information severely bottlenecks downstream processing. Early work by Joungbum Kim et al. in Detecting Structural Metadata with Decision Trees and Transformation-Based Learning and Yang Liu et al. in Comparing HMM, maximum entropy, and conditional random fields for disfluency detection established that when ASR systems fail to capture word fragments and subtle acoustic cues, downstream models cannot accurately detect where a speaker self-corrects. Word fragments serve as the primary structural signposts for restarts; omitting them deprives models of the signals needed to parse conversational repairs.
Furthermore, the choice of ASR architecture introduces distinct biases into the text. Maria Teleki et al. evaluated commercial systems in Comparing ASR Systems in the Context of Speech Disfluencies and found a stark trade-off: while some systems (like WhisperX) excel at capturing conversational interjections ("uh"), others are more faithful to structural restarts. When these elements are handled inconsistently or erased entirely, it introduces systemic bias. As Maria Teleki et al. argued in Quantifying the Impact of Disfluency on Spoken Content Summarization, treating disfluencies purely as noise risks creating incomplete or biased representations of speakers, particularly those with speech disorders.
Reclaiming the Lost Signal
To counteract this historical loss of data, the field has evolved along two main paths: preserving disfluencies at the source, or reconstructing them post-hoc.
Historically, datasets like SVitchboard 1, curated by Simon King 0001 et al. in SVitchboard 1: small vocabulary tasks from Switchboard, deliberately retained filled pauses and word fragments to force acoustic models to grapple with the structural irregularities of natural speech. When dealing with legacy datasets where disfluencies have already been scrubbed, researchers rely on architectures like "UNEDITOR," developed by Victoria Zayats et al. in Unediting: Detecting Disfluencies Without Careful Transcripts. This system uses data transformation techniques to detect and reconstruct "hidden" speech repairs and repetitions directly from edited text.
Ultimately, the shift toward disfluency-aware processing represents a deeper realization in spoken language understanding: human speech is not merely written text delivered orally. The hesitations, stammers, and corrections are an intricate, parallel channel of communication. By treating these elements as features rather than flaws, modern language models can bridge the gap between merely transcribing words and truly understanding human expression.
Key Foundational Work
Invited Talk: Processes that Shape Conversation and their Implications for Computational Linguistics (2000) — contributed 1 concept elements to this direction.
Processing Japanese Self-correction in Speech Dialog Systems (2002) — contributed 1 concept elements to this direction.