Lexical Repetition Patterns for Speech Repair Detection

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 220 papers29 conceptsfrozen 2026-10-04

When humans speak spontaneously, they do not produce the clean, linear sentences found in written text. Instead, we produce a "word soup" filled with hesitations, restarts, and self-corrections. A speaker might say, "A flight to—um—Boston, I mean, Denver on Tuesday." To a traditional natural language parser, this is noise. To a human listener, the structure is clear: "Boston" is being replaced by "Denver."

Research into speech repair detection focuses on identifying these "messy" segments and recovering the speaker's intended meaning. The core intuition driving this field is that speech repairs are not random errors; they follow a structured, almost mathematical logic of lexical repetition and structural copying.

The Copying Logic of Self-Correction

The fundamental observation, championed by researchers like Donald Hindle in Deterministic Parsing of Syntactic Non-fluencies, is that self-corrections often involve a "structured copying" of syntactic elements. When a speaker realizes they have made a mistake, they typically "rewind" to a previous point and restart the phrase.

This creates a predictable pattern: the reparandum (the error) and the repair (the correction) often share identical words or words from the same lexical category. Mark Johnson and Eugene Charniak formalized this in A TAG-based noisy-channel model of speech repairs, treating the repair as a "rough copy" of the intended correction. By identifying these modified duplicates, systems can reliably determine which segments to "expunge" to recover the fluent sentence.

The Power of Surface-Level Cues

While one might assume that understanding a self-correction requires deep semantic knowledge, evidence suggests that surface-level lexical patterns are surprisingly sufficient. Peter A. Heeman and James F. Allen demonstrated in Deyecting and Correcting Speech Repairs that local lexical clues—such as word repetitions and part-of-speech transitions—can resolve most common disfluencies without complex acoustic analysis or deep parsing.

Key mechanisms used to flag these repairs include:

  • Lexical Identity: Identifying exact word or bigram repetitions within a narrow temporal window (Mari Ostendorf and Sangyun Hahn).
  • Spatial Distance: Differentiating between a repair (usually immediate) and an intentional grammatical parallelism (usually spanning a longer sequence), as explored by Xian Qian and Yang Liu.
  • Contextual Similarity: Measuring word overlaps between the immediate left and right contexts of a potential interruption point (Narendra K. Gupta and Srinivas Bangalore).

In Mandarin Chinese, this repetition cue is even more dominant. Yuan Zhao and Daniel Jurafsky found that 90% of word fragments in Mandarin are characterized by lexical repetitions, making the identity of surrounding words the primary indicator for detection (Detection of word fragments in Mandarin telephone conversation).

Integrating Multi-Modal Signals

While lexical patterns provide high recall, they can be further refined by integrating other knowledge sources. John Bear and Elizabeth Shriberg showed that combining pattern matching with acoustic cues—like pitch resets or glottalization—helps distinguish between a speaker's errors and their intended meaning (Integrating Multiple Knowledge Sources for Detection and Correction of Repairs in Human-Computer Dialog).

Similarly, Julian Hough and Matthew Purver introduced the idea of using information-theoretic metrics like "surprisal." A localized drop in language model fluency often signals the onset of a repair, allowing for Strongly Incremental Repair Detection in real-time systems.

Efficient Architectures for Cleaning Speech

To make these insights practical for real-time Natural Language Understanding (NLU), researchers have developed specialized architectures. Tim Miller proposed using a short-term word buffer to align incoming speech with recent history, effectively creating a "memory" that catches repetitions as they happen (Word Buffering Models for Improved Speech Repair Parsing).

Modern approaches have moved toward transition-based parsing. Dongdong Zhang and colleagues demonstrated that processing text right-to-left more naturally captures the structure of human speech repairs, as the "correction" provides the necessary context to identify the "error" that preceded it (Efficient Disfluency Detection with Transition-based Parsing).

By treating disfluencies not as random noise, but as structured lexical events, these systems transform the "word soup" of human conversation into clean, actionable data for machines.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 24 papers
  2. Version 1 · 2026-10-04 · 20 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_surface_level_disfluency_detection_22807125,
  title = {surface-level disfluency detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/surface-level-disfluency-detection/rdr_22807125e5de43309468ddd9deee40b6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_lexical_repetition_patterns_for_speech_r_22807125_v1,
  title = {Lexical Repetition Patterns for Speech Repair Detection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/lexical-repetition-patterns-for-speech-repair-detection/rdr_22807125e5de43309468ddd9deee40b6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}