Abstract
Automatic Speech Recognition (ASR) systems frequently misinterpret acoustically similar words. Traditionally, analyzing these errors requires large-scale audio processing and manual verification. This paper explores "audio-free" error prediction, a technique that simulates recognition errors using only text and the mathematical definitions of a system's acoustic and pronunciation models. By integrating empirical error logs with Hidden Markov Model (HMM) distances within a Weighted Finite State Transducer (WFST) framework, the researchers demonstrate that up to 60% of sentence-level errors and 70% of local "error chunks" can be predicted in continuous speech.
Introduction
ASR systems are prone to confusing phonetically similar words, such as "accept" and "except." Understanding these failure modes is critical for developing robust language models and optimized lexicons. However, traditional error analysis is resource-intensive. Audio-free error prediction offers a scalable alternative, allowing developers to generate millions of synthetic errors to train more resilient systems. The primary challenge is accurately modeling a specific ASR system's "ears" without actual acoustic input.
Methodology
The researchers utilized a Weighted Finite State Transducer (WFST) framework to model the recognition process as a series of mathematical compositions: mapping words to pronunciations, pronunciations to phones, and phones to acoustic features.
A central component of this model is the Confusion Matrix ($C$), which quantifies the likelihood of phone-level misrecognitions. Four strategies for constructing $C$ were evaluated:
- FST-PhoneRec: Derived from raw phone recognition on training data.
- FST-WordRec: Derived from word recognition, incorporating lexical constraints.
- FST-HMM: Based on the Bhattacharya distance between phone HMMs.
- FST-Words+HMM: A hybrid approach combining empirical word-level logs with HMM distances.
Experimental Setup
The framework was tested on two benchmarks:
- NYNEX Phonebook: A corpus of isolated words used to evaluate acoustic and pronunciation models in isolation.
- Wall Street Journal (WSJ0): A continuous speech corpus used to test the framework's performance with full sentences and bigram language models.
Key Results and Discussion
The experiments revealed that the hybrid model consistently outperformed simpler empirical or purely mathematical approaches. As shown in Figure 4, the hybrid FST-Words+HMM approach achieves the highest fraction of predicted errors across various N-best list sizes compared to other confusion matrix strategies.
A significant finding was the impact of lexical constraints and language models. Lexical constraints act as a filter, narrowing down possible phone confusions to those that form valid words. In continuous speech (WSJ0), the inclusion of a Language Model (LM) was essential; without it, error prediction accuracy dropped significantly.
The researchers also introduced the concept of "error chunks"—local segments where the recognizer fails. While predicting full, long sentences is a high bar, predicting these local failures is more feasible and practically useful. On the WSJ0 task with a 16 GMM system, the model correctly predicted nearly 70% of these error chunks within the top 500 candidates.
Conclusion
This work demonstrates that audio-free error prediction is a viable tool for ASR development. By combining empirical data with acoustic distances, the proposed hybrid WFST model provides a granular understanding of system vulnerabilities. Future improvements may involve incorporating contextual information, such as co-articulation effects, to further refine the accuracy of predicted phone confusions.
Summary updated