Using Sub-word-level Information for Confidence Estimation with Conditional Random Field Models

Matthew Stephen Seigel, Philip C. WoodlandInterspeech 2012 · OpenReview

Introduction

In Automatic Speech Recognition (ASR), Confidence Estimation (CE) is the critical process of assigning a probability to a predicted word to indicate its likelihood of being correct. High-quality CE is vital for downstream tasks like dialogue management or metadata indexing. This paper explores how to improve word-level confidence estimation by incorporating "sub-word" information, such as individual characters or phonemes, using Conditional Random Fields (CRFs).

The Granularity Gap

Traditionally, CE relies on features extracted from word-level recognition lattices. However, ASR systems process speech at a much finer grain (sub-word units). The challenge, known as the "granularity gap," is that information at the sub-word level does not always align perfectly with word boundaries. This research addresses how to effectively integrate these rich, low-level acoustic and phonetic cues into a word-level confidence framework.

Methodology: CRF and HCRF Architectures

The authors propose two main architectural approaches using Conditional Random Fields, which are discriminative models well-suited for sequence labeling:

  1. Sub-word Linear-chain CRF: This model operates directly at the grapheme (character) level. To make it work for word-level CE, the label set is expanded to track both correctness and word boundaries (e.g., "Correct-Start," "Correct-Inside").
  2. Word-level Hidden-state CRF (HCRF): This model operates at the word level but introduces "hidden" variables. These hidden states allow the model to capture latent structures—such as "regions" of high or low confidence—without making the rigid independence assumptions found in older models like HMMs.
Comparison of Linear-chain CRF and CRF with Hidden states architectures

Key Feature Engineering

  • LSAPR (Lattice Sub-Arc Posterior Ratio): Calculating posterior probabilities for individual sub-word units directly from the ASR lattice.
  • MLP-based Features (AMP/WAMP): Using a separate, lightweight Multi-Layer Perceptron recognizer to provide a "second opinion." Because this MLP isn't constrained by a complex language model, its predictions are more purely tied to the acoustic evidence, making it a powerful error detector.
  • Word Boundary (WB) Functions: A specialized feature function that acts like a "skip-chain," allowing the sub-word model to maintain awareness of word-level transitions.

Experimental Results

The researchers evaluated their models using the Cambridge Arabic ASR system on GALE program data. Performance was measured using Normalized Cross Entropy (NCE)—where higher is better—and Utterance-level Mean Absolute Deviation (UMAD)—where lower is better.

Performance comparison (NCE and UMAD) of various confidence estimation systems
  • Sub-word CRF Improvements: On the difficult dev10d set, the best sub-word CRF (using MLP features and word boundary logic) achieved an NCE of 0.353, an 8.6% relative improvement over the decision-tree baseline (0.325).
  • Hidden-state CRF Gains: The word-level HCRF with two hidden states and sub-word features reached an NCE of 0.363, representing a 6.5% relative improvement over a standard word-level CRF.
  • Consistency: The inclusion of sub-word information consistently reduced the UMAD (error in confidence), dropping it from 12.64 in the baseline to 10.81 in the best HCRF model.

Conclusion and Future Work

The study demonstrates that acoustic "second opinions" from MLPs and the modeling of "confidence regions" via hidden states are highly effective for CE. While increasing the number of hidden states in the HCRF improved performance, the gains were modest compared to the increased complexity. The authors suggest that the next logical step is to apply the Hidden-state CRF directly at the sub-word level, potentially capturing even finer latent structures in the acoustic signal.

Summary updated

BibTeX
@inproceedings{seigel2012using,
  author = {Matthew Stephen Seigel and Philip C. Woodland},
  title = {{Using Sub-word-level Information for Confidence Estimation with Conditional Random Field Models}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=8JldZWQEjujF},
}
Figure