Enhancing the Robustness of the Posterior-Based Confidence Measures Using Entropy Information for Speech Recognition

Yanqing Sun, Yu Zhou, Qingwei Zhao, Pengyuan Zhang, Fuping Pan, Yonghong YanIEICE Trans. Information and Systems 2010 · OpenReview

Abstract

In this paper, the robustness of the posterior-based confidence measures is improved by utilizing entropy information, which is calculated for speech-unit-level posteriors using only the best recognition result, without requiring a larger computational load than conventional methods. Using different normalization methods, two posterior-based entropy confidence measures are proposed. Practical details are discussed for two typical levels of hidden Markov model (HMM)-based posterior confidence measures, and both levels are compared in terms of their performances. Experiments show that the entropy information results in significant improvements in the posterior-based confidence measures. The absolute improvements of the out-of-vocabulary (OOV) rejection rate are more than 20% for both the phoneme-level confidence measures and the state-level confidence measures for our embedded test sets, without a significant decline of the in-vocabulary accuracy.

Introduction

A significant challenge in real-world speech recognition is the "Out-of-Vocabulary" (OOV) problem. When a user utters a word not present in the system's lexicon, the recognizer typically attempts to match it to the closest known word, often resulting in a high-confidence but incorrect output. To mitigate this, systems employ Confidence Measures (CMs) to decide whether to accept or reject a recognition result.

While posterior-based measures are the industry standard, they often lack robustness when training and testing environments mismatch (e.g., noise or non-native accents). This research proposes enhancing these measures by incorporating entropy—a mathematical representation of uncertainty—to provide a more reliable "sanity check" for recognition outputs.

Methodology

The proposed approach introduces entropy information as a secondary layer of confidence. Unlike standard posterior measures that focus solely on the top candidate, entropy evaluates the distribution of probabilities across all possible speech units (phonemes or states).

Flowchart of the proposed confidence measure calculation incorporating background entropy information.

Entropy Calculation and Normalization

The system calculates the entropy ($H$) of the posterior probabilities. High entropy indicates the system is "confused" by multiple likely candidates, while low entropy suggests high certainty in a single candidate. To integrate this with standard scores, two normalization methods ($cm_{H1}$ and $cm_{H2}$) were developed. The final confidence score is a linear combination of the standard posterior and the normalized entropy in the logarithm domain, which effectively penalizes high-uncertainty results.

Granularity and Efficiency

The method was evaluated at two levels:

  • State-level: High accuracy but computationally intensive, involving individual HMM states.
  • Phoneme-level: Optimized for embedded devices, focusing on distinct speech sounds.

To maintain efficiency for low-power hardware, the researchers utilized a "maximum approximation" for likelihood summation, ensuring minimal computational overhead.

Experimental Setup

The evaluation utilized the THINKIT embedded speech recognition system. The training data consisted of clear, well-pronounced English. The test suite included five "In-Vocabulary" (IV) sets and three "Out-of-Vocabulary" (OOV) sets. These sets were designed to be challenging, featuring dialectal variations and non-native speech to simulate real-world environmental mismatches.

Key Results

The inclusion of entropy information yielded substantial performance gains. The OOV rejection rate improved by over 20% (absolute) at both the phoneme and state levels.

Comparison of False Rejection Rate (FRR) vs. False Alarm Rate (FAR) for baseline posterior, entropy-only, and combined confidence measures at the phoneme level.

As shown in the ROC curves, the Equal Error Rate (EER) for phoneme-level tests dropped from a baseline of 22.4% to approximately 6.7% when using the combined measure. Importantly, these improvements in rejecting OOV words did not compromise the accuracy of in-vocabulary word recognition, demonstrating the stability of the proposed method.

Conclusion

This research demonstrates that entropy provides critical "background context" that standard posterior probabilities miss. By combining these two sources of information in the log domain, speech recognition systems become significantly more robust to environmental mismatches and OOV inputs. While state-level analysis offers the highest precision, phoneme-level entropy represents a highly effective "sweet spot" for performance and efficiency in embedded applications. Future work may explore the application of these entropy-based measures to modern end-to-end deep learning architectures.

Summary updated

Abstract

In this paper, the robustness of the posterior-based confidence measures is improved by utilizing entropy information, which is calculated for speech-unit-level posteriors using only the best recognition result, without requiring a larger computational load than conventional methods. Using different normalization methods, two posterior-based entropy confidence measures are proposed. Practical details are discussed for two typical levels of hidden Markov model (HMM)-based posterior confidence measures, and both levels are compared in terms of their performances. Experiments show that the entropy information results in significant improvements in the posterior-based confidence measures. The absolute improvements of the out-of-vocabulary (OOV) rejection rate are more than 20% for both the phoneme-level confidence measures and the state-level confidence measures for our embedded test sets, without a significant decline of the in-vocabulary accuracy.
BibTeX
@article{sun2010enhancing,
  author = {Yanqing Sun and Yu Zhou and Qingwei Zhao and Pengyuan Zhang and Fuping Pan and Yonghong Yan},
  title = {{Enhancing the Robustness of the Posterior-Based Confidence Measures Using Entropy Information for Speech Recognition}},
  journal = {IEICE Trans. Information and Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=fwqu8n07Oq},
}
Figure