Prosodic word-based error correction in speech recognition using prosodic word expansion and contextual information

Chao-Hong Liu, Chung-Hsien WuInterspeech 2010

TL;DR

This paper proposes a novel approach to correct errors in spontaneous speech recognition by treating "prosodic words" (units related to rhythm and breath) as the fundamental correction units, rather than individual lexical words. By expanding these prosodic word candidates and leveraging contextual information, they achieve significant improvements in error correction accuracy compared to existing methods.

What Problem Does This Paper Address?

Spontaneous speech is notoriously difficult for Automatic Speech Recognition (ASR) systems, leading to a significant number of errors. While error correction techniques exist, they often struggle with the inherent variability and nuances of natural speech. Existing methods typically focus on correcting individual lexical words or rely on statistical machine translation (SMT) approaches. However, these methods can be computationally intensive or fail to capture the contextual dependencies that influence speech errors. This paper addresses the limitations of current ASR error correction by proposing a method that leverages prosodic words, which are argued to be more natural cognitive units of speech and better reflect how errors occur in spontaneous speech, especially due to phrasing and breathing patterns.

Methods

The core of the proposed method lies in using prosodic words as the primary units for error correction. Here's a breakdown of their approach:

  • Prosodic Word Extraction: They first identify prosodic word boundaries within speech. This is done by training a Classification and Regression Tree (CART) model to predict prosodic word breaks based on syllable boundaries and contextual features.
  • Mis-Recognized Word Fragment (MWF) Table: A crucial component is a table that maps recognized word fragments to potential prosodic words that might have been misrecognized. This table is built from a parallel corpus of correct and ASR-recognized sentences.
  • Prosodic Word Candidate Expansion: For each word fragment in a recognized sentence, the MWF table is used to retrieve a set of candidate prosodic words. This expands the possibilities beyond simple lexical word substitutions.
  • Contextual Information: To refine the correction, they incorporate two types of scores:
  • Substitution Score: This measures the likelihood of a specific prosodic word being misrecognized as a given word fragment.
  • Concatenation Score: This assesses the compatibility between adjacent prosodic words, considering their contextual information.
  • Probabilistic Model: These scores are combined within a probabilistic model to find the most likely sequence of corrected prosodic words, ultimately leading to the corrected text output.

Datasets

  • Spoken Dialogue Corpus: A custom corpus was collected for the travel information task, consisting of 144 dialogues and 1586 sentences from six male and two female subjects. This corpus was used for evaluating the proposed approach.
  • TH-COSS Corpus: This corpus was used for training the CART model for prosodic word extraction.

Key Insights and Ideas

  • Prosodic Words as Natural Units: The paper strongly advocates for prosodic words as more appropriate units for error correction than lexical words. This is because prosodic grouping (related to rhythm and breath) influences how speech is articulated and, consequently, how errors manifest.
  • Contextual Information is Key: Simply correcting individual word fragments isn't enough. The interaction and compatibility between adjacent prosodic words (concatenation score) are vital for producing fluent and accurate corrections.
  • Expansion Improves Candidate Pool: By expanding potential prosodic word candidates, the system can consider a wider range of corrections, moving beyond direct lexical substitutions and potentially capturing more complex misrecognitions.

Key Results

The proposed prosodic word-based error correction method demonstrated significant improvements:

  • F1 Score Improvement: Achieved a 0.32 F1 score, which is a substantial improvement of 0.18 over the SMT-based baseline and 0.10 over the lexical word-based approach.
  • BLEU Score: The proposed method achieved a BLEU score of 0.497, slightly outperforming the ASR recognized texts (0.491) and significantly outperforming the SMT-based method (0.437). This indicates that the corrections are generally beneficial rather than detrimental.
  • SMT Limitations: The study found that SMT-based correction, while promising in theory, underperformed due to insufficient training data and the tendency of n-gram models to prioritize fluency over accuracy in this specific ASR error correction context.
  • Lexical Word Limitations: The lexical word-based approach suffered from over-correction, applying too many unnecessary changes that lowered precision and recall.

Limitations and Future Work

  • SMT Performance: The authors note that the SMT approach's poor performance might be attributed to insufficient training data and the specific characteristics of ASR errors. They suggest that further investigation with more training data could be beneficial.
  • Generalizability of SMT: The paper implies that SMT might not be the most suitable solution for ASR error correction in general, especially when compared to methods that leverage linguistic and prosodic structures.
  • Future Work: While not explicitly detailed, the success of this prosodic word-based approach suggests future work could explore more sophisticated prosodic feature extraction, advanced contextual modeling, and integration with other linguistic information.

Summary updated

BibTeX
@inproceedings{liu2010prosodic,
  author = {Chao-Hong Liu and Chung-Hsien Wu},
  title = {{Prosodic word-based error correction in speech recognition using prosodic word expansion and contextual information}},
  booktitle = {Interspeech},
  year = {2010},
}

Loading related content...

Figure