When humans write, they generally adhere to a linear, prescriptive grammar. When humans speak spontaneously, however, that structure dissolves. Natural conversation is riddled with disfluencies: filled pauses (um, ah), repetitions (to the— to the store), false starts, and mid-sentence self-corrections (repairs).
While human listeners filter out this linguistic "noise" effortlessly, traditional Natural Language Processing (NLP) systems suffer from a severe "garbage in, garbage out" dilemma, as noted by Alex Waibel in Spontaneous Speech Consolidation for Spoken Language Applications. Models trained on formal, written text expect clean hierarchical trees. When confronted with the messy reality of spontaneous speech, their performance collapses.
For instance, Maheshwari et al. demonstrate in ASR Benchmarking: Need for a More Representative Conversational Dataset that Word Error Rates (WER) can triple or quadruple when moving to real-world human interactions. Even with perfectly transcribed speech, Szymanski et al. show in Why Aren't We NER Yet? Artifacts of ASR Errors in Named Entity Recognition in Spontaneous Speech Transcripts that sequence labeling models fail on nearly 40% of named entity spans due to conversational irregularities.
Why Disfluencies Break Traditional Parsers
To understand why spontaneous speech paralyzes standard parsers, consider how a statistical or dependency parser operates. It calculates the probability of grammatical relationships based on sequential word neighbors and hierarchical rules. Disfluencies break these local syntactic patterns in two primary ways:
- Sequence Disruption: Interjections and parentheticals insert arbitrary tokens between words that belong together grammatically. As Engel et al. explain in Parsing and Disfluency Placement, this drastically increases the variety of possible word neighbors, diluting the probability calculations used to determine grammatical relationships.
- Structural Non-Linearity: A speech repair or false start means a speaker abandons a syntactic path mid-sentence and restarts. Standard parsers try to force the entire utterance into a single coherent tree, leading to catastrophic parsing failures.
This structural mismatch propagates downstream. For example, Zheng et al. demonstrate in Using syntax in large-scale audio document translation that while syntax-aware models significantly boost machine translation quality for structured text, they provide zero benefit for broadcast conversations because the underlying parsers cannot resolve the ungrammatical, fragmented nature of dialogue.
The Two-Pass Architecture: Strip and Parse
The most intuitive engineering solution to this problem is a pipeline that decouples disfluency handling from syntactic parsing. Instead of forcing a parser to understand disfluencies, a specialized pre-processing classifier identifies and removes them, passing a "cleaned" string to a standard parser.
This approach was pioneered by Charniak et al. in Edit Detection and Parsing for Transcribed Speech. Their two-pass architecture first detects "speech repairs" (the words to be deleted) and then runs a standard statistical parser over the remaining text. To prevent the evaluation from being overly punitive, the authors also introduced a relaxed metric that does not penalize the parser for the exact placement of edited words within the discarded structure.
Similarly, Bear et al. in Integrating Multiple Knowledge Sources for Detection and Correction of Repairs in Human-Computer Dialog show that combining pattern matching with syntactic and semantic parsing allows systems to automatically detect and fix self-corrections without needing an explicit "edit signal."
Leveraging Multi-Modal and Acoustic Cues
Text transcripts hide critical clues that humans use to navigate disfluent speech. When a speaker stutters or changes their mind, they don't just change their words; they alter their pitch, pause, or change their physical gestures. Integrating these extra-linguistic signals provides vital context for models attempting to locate speech repairs.
Acoustic-Prosodic Cues
Features like pitch resets, duration, and energy changes signal where a speaker is hesitating or correcting themselves. Tran et al. demonstrate in On the Role of Style in Parsing Speech with Neural Models that while large pre-trained language models like BERT provide a strong baseline, incorporating prosodic information offers critical performance gains when parsing long, highly disfluent sentences.
Visual Gestures
Hand movements can act as physical markers of discourse structure. Jacob Eisenstein demonstrates in Semantic Back-Pointers from Gesture that identifying when a speaker repeats a gesture or returns their hand to a specific spatial location serves as a "semantic back-pointer." This physical cue helps systems resolve coreferences and detect speech restarts more accurately than text-only models.
Flat and Sequential Representations
An alternative to the two-pass pipeline is to abandon rigid, deeply hierarchical grammars altogether. If spontaneous speech is fundamentally non-linear, forcing it into a strict hierarchical tree is bound to fail. Instead, researchers have developed "flat" or sequential architectures that explicitly accommodate conversational irregularities.
Wermter et al. introduced this philosophy with the SCREEN system in SCREEN: Learning a Flat Syntactic and Semantic Spoken Language Analysis Using Artificial Neural Networks and Learning Fault-Tolerant Speech Parsing with SCREEN. By utilizing neural networks to map speech to shallow sequences of categories rather than deep hierarchical trees, the system achieves fault tolerance against stutters, repairs, and speech recognition errors.
In a similar vein, sequential labeling techniques can be adapted to treat disfluencies as valid structural elements rather than noise. Kenji Imamura demonstrates in Japanese Dependency Parsing Using Sequential Labeling for Semi-spoken Language that by using Conditional Random Fields (CRFs) and introducing a specialized "self-dependency" category, a parser can gracefully handle conversational fillers and markers that would otherwise break traditional dependency relations. This shift from eliminating disfluencies to integrating them into the sequence allows models to preserve the full conversational context.