TL;DR
This paper introduces a novel, "lexically-driven" algorithm for automatically detecting disfluencies (like "um," "uh," and speech repairs) in transcribed speech. Surprisingly, it achieves performance comparable to systems heavily reliant on complex acoustic features, demonstrating that rich linguistic information alone can be highly effective. The approach also shows promise in adapting to speaker styles to better identify less common disfluency markers.
What Problem Does This Paper Address?
Human speech is full of "disfluencies" – speech repairs, filler words ("um," "uh"), and repetitions – that significantly hinder downstream natural language processing tasks (like speech recognition and parsing) and make reading transcripts difficult. While many researchers have focused on using acoustic cues (pitch, tone, pauses) to detect these disfluencies, this paper explores a different path: can we achieve good performance using only the words themselves and their grammatical roles (part-of-speech tags)? This is crucial because acoustic data is often unavailable or noisy in real-world scenarios, and a purely lexical approach could be more robust and efficient.
Methods
The core of this work is a Transformation-Based Learning (TBL) algorithm, a technique that learns a set of rules to transform an initial hypothesis (in this case, labeling all words as "fluent") into a more accurate prediction. The system, dubbed "System A," uses a rich set of lexical features:
- Lexeme: The actual word itself.
- Part-of-Speech (POS) tag: The grammatical category of the word.
- Speaker Style Feature: Whether a word is unusually frequent for a specific speaker compared to the general corpus. This is a clever way to catch words like "like" that are often fluent but can signal disfluency when overused by a particular speaker.
The TBL algorithm iteratively learns rules that best reduce errors, starting from the assumption that all words are fluent and then learning to relabel them as "edits" (repairs) or "fillers."
Datasets
The research utilizes datasets from the Linguistic Data Consortium (LDC) for the EARS (Effective, Affordable, Reusable Speech-to-Text) program.
- Training Data:
- Conversational Telephone Speech (CTS): 491,543 tokens
- Broadcast News Speech (BNEWS): 189,766 tokens
- Evaluation Data:
- CTS: 33,670 tokens
- BNEWS: 14,544 tokens
The evaluation data was held out and not used for tuning system parameters.
Key Insights and Ideas
- Lexical Powerhouse: The most striking insight is that a system relying primarily on lexical features can perform comparably to systems that heavily incorporate prosodic (acoustic) cues. This challenges the assumption that acoustic information is indispensable for accurate disfluency detection.
- Speaker Style Matters: The introduction of a "high frequency word" feature, which captures words used unusually often by a specific speaker, proved valuable. This allows the system to identify less common disfluency patterns that might be missed by a general model.
- TBL's Adaptability: The TBL framework's ability to learn a compact set of rules from data makes it a promising candidate for extending disfluency detection to other languages, as it doesn't require massive training corpora.
Key Results
System A demonstrated competitive performance against two other systems (System B and System C) that utilized prosodic features.
- Comparable Performance: For most conditions, System A's disfluency detection results were similar to or slightly better than Systems B and C.
- Speech Output Challenges: The paper highlights that errors in the Speech-to-Text (STT) output significantly impact disfluency detection. For instance, the filler error rate in broadcast news speech jumped from 6.5% (reference transcript) to 57.2% (STT output) for System A, indicating that STT errors are a major bottleneck.
- Edit vs. Filler Difficulty: System A struggled more with detecting "edits" than "fillers," particularly in longer edit regions.
Limitations and Future Work
- Longer Edits: The system still misses longer edits (four words or more), which are a significant source of errors.
- STT Error Robustness: The performance is heavily degraded by errors in the Speech-to-Text output. The authors suggest training on STT output directly or incorporating word recognition confidence scores as potential solutions.
- Prosodic Cues for Edits: For identifying the boundaries of longer edits, prosodic features (like interruption points) might eventually be necessary.
- Cross-Lingual Extension: The authors believe the TBL approach is well-suited for extending disfluency detection to other languages due to its efficiency with training data.
- Parsing Integration: Future work could involve integrating parsing techniques to better identify longer edits and using parse tree structures as features.
Summary updated