Improved word confidence estimation using long range features

David D. Palmer, Mari OstendorfInterspeech 2001 · DOI

TL;DR

This paper introduces novel "long-range" features that significantly improve word confidence estimation in speech recognition systems. By incorporating information from the entire document and task-specific statistics, these improved confidences lead to better performance in downstream information extraction tasks, particularly named entity recognition.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often output a "confidence score" for each recognized word, indicating the probability that the word is correct. These scores are crucial for various applications, such as filtering out errors in dialogue systems or improving information extraction (IE) from transcribed speech. However, standard confidence estimation methods often rely on local features around a word, potentially missing valuable context. This paper addresses the limitation of existing confidence estimation techniques by exploring the use of "long-range" features that leverage broader contextual information, aiming to provide more accurate confidence scores and, consequently, enhance the performance of IE systems.

Methods

The authors propose augmenting standard ASR confidence predictors with two types of novel features:

  • Document-level features: These features capture information from a wider window around a hypothesized word, potentially encompassing the entire document. Examples include the confidence scores of neighboring words (e.g., $y_{t-1}, y_{t+1}$) and ratios of a word's confidence to the average confidence of the entire document. The intuition is that patterns of low confidence scores in sequence, or a word's confidence relative to the overall document's confidence, can be indicative of errors.
  • Task-dependent features: These features are derived from the entire training corpus and reflect the overall performance of the ASR system. Specifically, they use the relative frequency of a word (and its neighbors) occurring as an error in the training data. This helps to identify words that are inherently prone to being misrecognized.

They experimented with three estimation techniques:

  • Decision Trees (DT): A standard machine learning approach for classification and regression, used here to partition the data based on features and assign confidence scores based on the proportion of correct/error words in leaf nodes.
  • Generalized Linear Models (GLM): A log-linear model that linearly combines features with learned weights to predict confidence.
  • Interpolation of DT and GLM: Combining the outputs of the DT and GLM models to leverage their complementary strengths.

Datasets

The paper does not explicitly name the datasets used for training and evaluation. However, it mentions using a "development test data" for tuning and refers to a "baseline confidence predictor we have worked with (from Dragon)" and a "Dragon Systems' Automatic Transcription of New TDT Corpus" in the references, suggesting the use of speech recognition output from Dragon Systems, likely on a corpus related to Topic Detection and Tracking (TDT).

Key Insights and Ideas

  • Context is King (Even for Confidence): The most significant insight is that extending the context beyond immediate neighbors dramatically improves word confidence estimation. Document-level and task-specific statistics provide crucial information that local features miss.
  • Errors Often Come in Batches: The observation that neighboring word confidences ($y_{t-1}, y_{t+1}$) are useful suggests that speech recognition errors are not always isolated events but can occur in sequences.
  • Task-Specific Biases Matter: Understanding how often a word is misrecognized in general within the training data (task-dependent features) is a powerful indicator of its potential for error in new data.
  • IE Benefits Directly: The improved confidence scores aren't just an academic exercise; they directly translate to tangible improvements in downstream tasks like Named Entity Recognition (NER), demonstrating the practical value of better confidence estimation.
  • Multi-Pass Processing Pays Off: Iteratively re-estimating confidences using information from an IE system (like NER) and then using those improved confidences to further refine the IE system creates a beneficial feedback loop.

Key Results

  • The proposed long-range features (document-level and task-dependent) improved Normalized Cross Entropy (NCE) from a baseline of 0.195 to 0.262.
  • The interpolated DT and GLM model achieved the best confidence estimation performance with an NCE of 0.262.
  • Incorporating named entity information as an additional feature further boosted NCE to 0.287.
  • These improved confidences led to a significant reduction in the Slot Error Rate (SER) for Named Entity Recognition, decreasing it by approximately 10% (from 46.4% to 34.4% for the best interpolated model).
  • A multi-pass experiment showed that using NER output to re-estimate confidences improved NCE and NER performance iteratively, with diminishing returns after a few passes.

Limitations and Future Work

  • Dataset Specificity: The performance gains are demonstrated on a specific dataset (likely related to TDT). It's unclear how well these improvements generalize to different acoustic conditions, languages, or domains.
  • Feature Engineering Effort: While effective, the proposed features require careful design and computation, potentially adding overhead to the system.
  • "Perfect" Confidences: The paper notes that even with significant improvements, the confidence scores are still far from "perfect" (where correct words have confidence 1 and errors have confidence 0), indicating room for further research.
  • Future Work: The authors suggest exploring the use of name information to further improve confidence estimation, as demonstrated in their multi-pass experiments. They also implicitly point towards investigating more sophisticated feature combinations and estimation techniques.

Summary updated

BibTeX
@inproceedings{palmer2001improved,
  author = {David D. Palmer and Mari Ostendorf},
  title = {{Improved word confidence estimation using long range features}},
  booktitle = {Interspeech},
  year = {2001},
  doi = {10.21437/eurospeech.2001-499},
}
Figure