Abstract
In this paper, the robustness of the posterior-based confidence measures is improved by utilizing entropy information, which is calculated for speech-unit-level posteriors using only the best recognition result, without requiring a larger computational load than conventional methods. Using different normalization methods, two posterior-based entropy confidence measures are proposed. Practical details are discussed for two typical levels of hidden Markov model (HMM)-based posterior confidence measures, and both levels are compared in terms of their performances. Experiments show that the entropy information results in significant improvements in the posterior-based confidence measures. The absolute improvements of the out-of-vocabulary (OOV) rejection rate are more than 20% for both the phoneme-level confidence measures and the state-level confidence measures for our embedded test sets, without a significant decline of the in-vocabulary accuracy.
Introduction
A significant challenge in real-world speech recognition is the "Out-of-Vocabulary" (OOV) problem. When a user utters a word not present in the system's lexicon, the recognizer typically attempts to match it to the closest known word, often resulting in a high-confidence but incorrect output. To mitigate this, systems employ Confidence Measures (CMs) to decide whether to accept or reject a recognition result.
While posterior-based measures are the industry standard, they often lack robustness when training and testing environments mismatch (e.g., noise or non-native accents). This research proposes enhancing these measures by incorporating entropy—a mathematical representation of uncertainty—to provide a more reliable "sanity check" for recognition outputs.
Methodology
The proposed approach introduces entropy information as a secondary layer of confidence. Unlike standard posterior measures that focus solely on the top candidate, entropy evaluates the distribution of probabilities across all possible speech units (phonemes or states).
Entropy Calculation and Normalization
The system calculates the entropy ($H$) of the posterior probabilities. High entropy indicates the system is "confused" by multiple likely candidates, while low entropy suggests high certainty in a single candidate. To integrate this with standard scores, two normalization methods ($cm_{H1}$ and $cm_{H2}$) were developed. The final confidence score is a linear combination of the standard posterior and the normalized entropy in the logarithm domain, which effectively penalizes high-uncertainty results.
Granularity and Efficiency
The method was evaluated at two levels:
- State-level: High accuracy but computationally intensive, involving individual HMM states.
- Phoneme-level: Optimized for embedded devices, focusing on distinct speech sounds.
To maintain efficiency for low-power hardware, the researchers utilized a "maximum approximation" for likelihood summation, ensuring minimal computational overhead.
Experimental Setup
The evaluation utilized the THINKIT embedded speech recognition system. The training data consisted of clear, well-pronounced English. The test suite included five "In-Vocabulary" (IV) sets and three "Out-of-Vocabulary" (OOV) sets. These sets were designed to be challenging, featuring dialectal variations and non-native speech to simulate real-world environmental mismatches.
Key Results
The inclusion of entropy information yielded substantial performance gains. The OOV rejection rate improved by over 20% (absolute) at both the phoneme and state levels.
As shown in the ROC curves, the Equal Error Rate (EER) for phoneme-level tests dropped from a baseline of 22.4% to approximately 6.7% when using the combined measure. Importantly, these improvements in rejecting OOV words did not compromise the accuracy of in-vocabulary word recognition, demonstrating the stability of the proposed method.
Conclusion
This research demonstrates that entropy provides critical "background context" that standard posterior probabilities miss. By combining these two sources of information in the log domain, speech recognition systems become significantly more robust to environmental mismatches and OOV inputs. While state-level analysis offers the highest precision, phoneme-level entropy represents a highly effective "sweet spot" for performance and efficiency in embedded applications. Future work may explore the application of these entropy-based measures to modern end-to-end deep learning architectures.
Summary updated