Improving Information Extraction by Modeling Errors in Speech Recognizer Output

David D. Palmer, Mari OstendorfHLT 2001 · OpenReview

Abstract

In this paper, we describe a technique for improving the performance of an information extraction system for speech data by explicitly modeling the errors in the recognizer output. The approach combines a statistical model of named entity states with a lattice representation of hypothesized words and errors annotated with recognition confidence scores. Additional refinements include the use of multiple error types, improved confidence estimation, and multipass processing. In combination, these techniques improve named entity recognition performance over a text-based baseline by 28%.

Introduction

Information Extraction (IE) systems are typically designed for clean, written text. However, when applied to spoken language—such as broadcast news or telephone conversations—performance degrades significantly due to errors introduced by Automatic Speech Recognition (ASR) systems. ASR output often contains insertions, deletions, and substitutions that standard IE systems treat as factual input. This paper proposes a probabilistic framework that explicitly models ASR uncertainty, allowing the IE system to weigh hypothesized words against the likelihood of recognition errors.

Methodology

The proposed system treats Information Extraction as a path-finding problem through a word lattice. The core components include:

  1. Statistical IE Model: A hidden Markov-like model using state-dependent bigrams to predict sequences of information states (e.g., Person, Location, None).
  2. Error Lattice Construction: Instead of a linear word sequence, the system constructs a lattice where each hypothesized word is paired with a parallel "error" arc, represented by an $\epsilon$-token. This allows the decoder to "skip" words it believes are misrecognized.
  3. Confidence-Weighted Decoding: The choice between a word path and an error path is guided by word confidence scores. Low confidence in a word increases the probability that the system will traverse the $\epsilon$ arc.
Lattice representation with explicit error modeling using $\epsilon$-tokens.
  1. Multi-Pass Refinement: Recognizing that names are often Out-of-Vocabulary (OOV) and thus prone to ASR errors, the authors implemented a multi-pass approach. An initial pass identifies potential named entities, which are then used as features to refine word confidence scores for a subsequent, more accurate decoding pass.

Experimental Evaluation

The system was evaluated using 114 news broadcasts from the 1998 DARPA Hub-4 Topic Detection and Tracking (TDT) evaluation. The ASR engine used a 57,000-word vocabulary and exhibited a high Word Error Rate (WER) of approximately 30%.

Performance was measured using the F-measure and Slot Error Rate (SER). As shown in the results table, the inclusion of error modeling and refined confidence scores consistently improved performance over the baseline.

Performance comparison of Named Entity recognition across different error modeling configurations.

Key Results

  • Baseline Performance: Without error modeling, the system achieved a 68.4 F-measure.
  • Full System Performance: Integrating multi-pass processing and name-aware confidence features raised the F-measure to 73.4.
  • Gap Recovery: Given the 30% WER, the theoretical maximum F-measure was 86.4. The proposed techniques recovered 28% of the potential performance gap caused by ASR errors.
  • Confidence Estimation: The new feature set improved the Normalized Cross Entropy (NCE) of word confidence prediction from 0.195 to 0.287.

Discussion and Conclusion

A critical insight of this work is the strong correlation between named entities and ASR failures; 57% of OOV errors occur within named entities. By modeling these errors explicitly and using a multi-pass strategy where IE results inform error detection, the system creates a virtuous cycle of improvement.

Future work may involve moving beyond the "1-best" ASR output to full lattice decoding, which would allow the IE system to consider multiple word hypotheses simultaneously. Additionally, while the current system excels at ignoring errors, future iterations could attempt to phonetically correct misrecognized words to further improve recall.

Summary updated

Abstract

In this paper we describe a technique for improving the performance of an information extraction system for speech data by explicitly modeling the errors in the recognizer output. The approach combines a statistical model of named entity states with a lattice representation of hypothesized words and errors annotated with recognition confidence scores. Additional refinements include the use of multiple error types, improved confidence estimation, and multipass processing. In combination, these techniques improve named entity recognition performance over a text-based baseline by 28%.
BibTeX
@inproceedings{palmer2001improving,
  author = {David D. Palmer and Mari Ostendorf},
  title = {{Improving Information Extraction by Modeling Errors in Speech Recognizer Output}},
  booktitle = {HLT},
  year = {2001},
  url = {https://openreview.net/forum?id=H1NJnMWOWH},
}
Figure