LM Studies on Filled Pauses in Spontaneous Medical Dictation

Jochen PetersNAACL 2003

TL;DR

This paper investigates how to best handle "filled pauses" (like "uh" or "um") in language models for spontaneous medical dictation. They found that removing filled pauses from the language model's history significantly improves prediction accuracy, suggesting these pauses are more often hesitations than restarts. This work is crucial for improving the accuracy of speech recognition systems in real-world, unscripted scenarios.

What Problem Does This Paper Address?

Spontaneous speech, especially in tasks like medical dictation, is rife with disfluencies. Filled pauses (FPs) are a common type, and they pose a significant challenge for Automatic Speech Recognition (ASR) systems. FPs can degrade ASR performance by introducing unexpected word sequences and by being acoustically similar to short function words. While previous research has explored FP handling in dialogues, this paper focuses on spontaneous medical dictation, a monologue-based task. The core questions are: 1) What's the optimal way to incorporate FPs into a language model's history, and 2) Can language models effectively distinguish between positions where FPs are likely versus unlikely?

Methods

The authors explore three main approaches for treating FPs in trigram language models (LMs):

  1. Treat FP as a regular word: FPs are predicted by the LM and condition subsequent words.
  2. Discard FP from history: FPs are still predicted, but they are removed from the LM's conditioning history for subsequent words. This assumes that the sentence continues after the FP.
  3. Fixed FP probability (unigram): A constant, low probability is assigned to FPs, and words are predicted using an FP-free LM history. This approach simplifies things by not requiring FP-tagged corpora.

They also employ "merged counts," which combine n-grams from corpora with and without FPs to leverage knowledge of both fluent and disfluent transitions.

Datasets

The experiments utilize approximately 1.4 million words of real-life dictated medical reports from US hospitals. This dataset is partitioned into training, development (Dev), and evaluation (Eval) sets. A separate "Report" corpus of non-spontaneous reports is also used for LM training. The spontaneous dictation data includes FP annotations, but lacks tags to distinguish between hesitations, repairs, or restarts. A key characteristic noted is that most FPs in this corpus are sentence-internal (70-80%), unlike in dialogues.

Key Insights and Ideas

  • FPs are mostly hesitations, not restarts: The results strongly suggest that FPs in medical dictation primarily signal a speaker searching for the right words or formulation, rather than a complete sentence break or restart. This is evidenced by the significant performance improvements when FPs are removed from the LM history.
  • Discarding FPs is beneficial: Removing FPs from the LM's conditioning history leads to substantial reductions in perplexity, particularly for words immediately following an FP.
  • Merged counts add value: Combining n-gram statistics from corpora with and without FPs further enhances LM performance, indicating that both types of transitions are informative.
  • LMs can discriminate FP likelihood: The developed LMs can effectively lower the probability of FPs at word positions, reducing confusion between FPs and actual words.

Key Results

  • Overall perplexity reduction: Discarding FPs from the LM history reduces overall perplexity by 4-5%.
  • Perplexity reduction after FPs: This reduction is much more significant (30-40%) for positions immediately following FPs.
  • Merged counts yield further gains: Merged counts provide an additional 3% reduction in overall perplexity and a 10% reduction in perplexity after FPs.
  • Trigrams outperform bigrams after FPs: This supports the idea that FPs are rarely sentence breaks.
  • FP probability discrimination: The LMs can reduce the FP probability at word positions by nearly 50%, which is crucial for distinguishing FPs from words.
  • ASR improvement: Speech recognition experiments show a relative error rate reduction of 2.2% on the Eval set when using the improved LMs.

Limitations and Future Work

The paper acknowledges that the spontaneous dictation data lacks detailed annotation for the type of disfluency (hesitation, repair, restart). This limits the granularity of their analysis. Future work could involve more detailed annotation to further refine FP handling strategies. The authors also suggest that their findings on FP prediction can be directly applied to improve ASR systems, as demonstrated by the reported error rate reduction.

Summary updated

BibTeX
@inproceedings{peters2003lm,
  author = {Jochen Peters},
  title = {{LM Studies on Filled Pauses in Spontaneous Medical Dictation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}

Loading related content...

Figure