Improving ASR error detection with non-decoder based features

Thomas Pellegrini, Isabel TrancosoInterspeech 2010

TL;DR

This paper tackles the challenge of accurately identifying errors in automatic speech recognition (ASR) outputs. The authors introduce novel features derived from external knowledge sources, like web search engines and topic models, to complement traditional decoder-based features. Their findings show that these new features significantly improve error detection accuracy, particularly when combined with powerful statistical models like Maximum Entropy and Conditional Random Fields.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems, while increasingly sophisticated, still make errors. These errors can manifest as substitutions (wrong word recognized), insertions (extra word recognized), or deletions (word missed). For applications like real-time subtitling or language learning tools, it's crucial to identify these errors to ensure the accuracy and usability of the recognized text. Traditional methods for error detection often rely on features directly from the ASR decoder (e.g., acoustic scores, language model probabilities). However, these features alone have limitations, as there's a significant overlap between correct and incorrect word hypotheses. This paper aims to bridge this gap by exploring new, "non-decoder based" features that can provide complementary information to improve the robustness of ASR error detection.

Methods

The core of this paper lies in the exploration of novel features for error detection, which are then fed into discriminative statistical models.

Novel Features:

  • Binary Word Match Feature ('w'): This feature compares the output of two ASR systems. One is the primary, "strong" system, and the other is a "weaker" system (e.g., using monophone acoustic models instead of context-dependent ones). If the words at the same position in the two transcripts differ, this feature is set to 0, indicating a potential error. This approach avoids the need for complex phone-to-word transducers.
  • Bigram Hit Feature ('h'): This feature leverages the power of web search engines. For each hypothesized bigram (a pair of consecutive words), queries are sent to a popular search engine. The number of "hits" (occurrences of the bigram) is retrieved. These raw hit counts are then quantized into discrete values (0 to 1) using heuristic rules. The intuition is that rare or non-existent bigrams in web content are likely to contain recognized errors.
  • Topic Feature ('t'): This feature aims to capture the semantic coherence of a hypothesized sentence. It uses unsupervised topic models (like Latent Dirichlet Allocation, LDA) to represent words and sentences as vectors of topic weights. A cosine similarity measure between a word's topic vector and its sentence's topic vector is used. A high similarity suggests the word fits the sentence's topic, potentially indicating higher confidence.

Statistical Models:

The paper evaluates these features using two discriminative models:

  • Maximum Entropy (Maxent) Models: A well-established technique for classification tasks, Maxent models learn to combine features by estimating optimal weights.
  • Linear-Chain Conditional Random Fields (CRFs): CRFs are sequential models that can capture dependencies between adjacent labels, making them suitable for sequence data like speech recognition outputs. They are presented as a sequential extension of Maxent models.

Datasets

  • Corpus: The study uses a subset of the ALERT European Portuguese Broadcast News (BN) corpus. This corpus consists of 14 manually transcribed TV newscasts from 2001, totaling approximately 14 hours of speech. The two most recent newscasts were reserved for testing.
  • ASR System: The material was automatically transcribed using an in-house ASR system called AUDIMUS, a hybrid Artificial Neural Networks / Hidden Markov Models system. The word error rate for the test subset was reported as 15.61%.
  • Training Data: For training the ASR system and topic models, a larger corpus was used, including the training portion of the ALERT corpus and an additional 31 million words of Portuguese newspaper text from 2001.

Key Insights and Ideas

  • External Knowledge is Powerful: The most significant takeaway is that leveraging information outside the direct ASR decoding process (like web search results and topic coherence) can substantially boost error detection performance. This challenges the notion that decoder-based features are sufficient.
  • Web Search as a Linguistic Oracle: The "Bigram Hit Feature" demonstrates the surprising effectiveness of using a general-purpose web search engine as a proxy for linguistic plausibility and correctness. The sheer volume and up-to-date nature of web content make it a valuable, albeit indirect, source of information.
  • Simplicity Can Be Effective: The "Binary Word Match Feature" shows that even a simple comparison between two ASR systems with slightly different acoustic models can yield significant improvements, without requiring complex new modeling components.
  • Model Choice Matters: While new features are important, the choice of classifier also plays a crucial role. Linear-chain CRFs, in this study, generally outperformed Maxent models, especially when combined with the best feature sets.
  • Topic Models Need Careful Application: The "Topic Feature" did not yield improvements, suggesting that applying unsupervised topic models at the word level might be challenging. The authors hypothesize that these models might be more effective at the document level.

Key Results

The paper reports significant improvements in Classification Error Rate (CER) and Detection Cost Function (minDCF) compared to a baseline using only decoder-based features.

  • Baseline: The baseline Maxent model achieved a CER of 13.87%, and the baseline CRF achieved 14.01%.
  • Best Performance (Maxent): Combining the baseline features with the binary word match ('w') and bigram hit ('h') features (Mwh) resulted in the best CER for Maxent, dropping to 12.16%. This represents a 12.3% relative reduction from the baseline CER.
  • Best Performance (CRF): Adding the binary word match ('w') feature to the baseline CRF (Cw) achieved a CER of 12.39%, a 10% relative reduction from the baseline CRF CER.
  • DET Curves: Figure 1 shows DET curves, visually confirming that the new feature sets (especially 'w' and 'h') lead to better trade-offs between miss probability and false alarm probability compared to the baseline.

Limitations and Future Work

  • Topic Feature Underperformance: The authors acknowledge that the topic feature did not improve performance. They suggest that this might be due to the difficulty of applying unsupervised topic models at the word level, and that more work is needed to explore their potential, perhaps at the document level.
  • One-Best Hypothesis: Experiments were conducted solely on the "one-best" hypothesis (the single most likely word sequence from the ASR). The authors plan to validate their approach by extending it to word lattices or confusion networks, which represent multiple hypotheses.
  • Limited Feature Sets: Future work will involve incorporating other features, such as local entropy, to account for competing word hypotheses.
  • Semantic Features: Further exploration of semantic-based features is also suggested.

Summary updated

BibTeX
@inproceedings{pellegrini2010improving,
  author = {Thomas Pellegrini and Isabel Trancoso},
  title = {{Improving ASR error detection with non-decoder based features}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure