TL;DR
This paper introduces e-WER2, a novel "no-box" approach to estimate the Word Error Rate (WER) of Automatic Speech Recognition (ASR) systems without needing access to the ASR system itself or its outputs. By leveraging a multistream end-to-end architecture that combines acoustic, lexical, and phonotactic features, e-WER2 can predict sentence-level WER, offering a more efficient and cost-effective way to assess ASR performance.
What Problem Does This Paper Address?
Evaluating the performance of Automatic Speech Recognition (ASR) systems typically relies on calculating the Word Error Rate (WER) against manually transcribed reference data. This process is notoriously time-consuming and expensive, especially for large datasets, as it requires hours of human annotation. While previous work has explored estimating WER using various features, this paper tackles the challenge of performing this estimation without any access to the ASR system's internal workings (glass-box), its outputs (black-box), or even the ASR system itself (no-box). This "no-box" scenario is particularly valuable for scenarios where one might only have access to audio and needs to gauge ASR quality without the original ASR pipeline.
Methods
The core of the e-WER2 system is a multistream end-to-end architecture. This architecture takes multiple types of features as input and jointly learns to predict the WER. The key innovation lies in the types of features used and the "no-box" capability:
- Feature Streams:
- Lexical (L): Word sequences derived from a standard word-based LVCSR system.
- Phonotactic (P): Sequences of phonemes extracted using phoneme recognizers. The authors found that using both Arabic and a general-purpose Hungarian phoneme recognizer (empirically shown to be multilingual) yielded robust phonotactic patterns.
- Acoustic (A): Mel-Frequency Cepstral Coefficients (MFCCs) extracted from audio segments.
- Decoder (D): Numerical features from the ASR decoder, such as frame counts and log-likelihoods.
- "No-Box" Approach: The most significant contribution is the "no-box" system. This variant learns a joint acoustic-lexical representation from phoneme recognition results and MFCC acoustic features to estimate WER. Crucially, it does not require access to the ASR system's internal decoder features or even its output transcriptions.
- Modeling:
- Numerical features are processed by a feed-forward neural network.
- Acoustic features are handled by deep Convolutional Neural Networks (CNNs).
- Lexical and phonotactic features are processed by CNNs, with specific input length constraints and embedding layers.
- All these streams are then concatenated and fed into a final fully connected layer to predict the sentence-level WER.
Datasets
The paper utilizes data from the Multi-Genre Broadcast (MGB-2) challenge for training and development. This dataset consists of Arabic broadcast programs from Aljazeera TV.
- Training/Development Data: Approximately 1,200 hours of audio.
- Evaluation Data: 10 hours of audio for development and 10 hours for evaluation, distinct from the training set.
- Test Set: A three-hour test set collected by BBC Media Monitoring from different broadcasters in November 2016, used to assess generalization to a different source.
The paper also mentions using background text from the Aljazeera Arabic website for language modeling.
Key Insights and Ideas
- "No-Box" WER Estimation is Feasible: The most striking insight is that WER can be effectively estimated without any access to the ASR system's internal states or outputs. This opens up new possibilities for ASR quality assessment in diverse scenarios.
- Phonotactic Features are Powerful: The inclusion of phonotactic features, derived from phoneme recognition, significantly improves WER estimation, especially in the "no-box" scenario. This suggests that the sequential patterns of sounds carry valuable information about speech recognition errors.
- Multistream Fusion Works: Combining diverse feature types (acoustic, lexical, phonotactic, decoder) in a multistream architecture leads to more robust and accurate WER estimation compared to using individual feature sets.
- Generalization to Different Sources: The system demonstrates reasonable performance even when tested on data from a different source (BBC Media Monitoring) than its training data (Aljazeera), indicating good generalization capabilities.
Key Results
- Overall WER Estimation: The proposed e-WER2 system achieved an overall estimated WER of 30.9% on a three-hour test set, closely matching the reference WER of 28.5%.
- Sentence-Level Performance:
- The "glass-box" version of e-WER2 achieved a Pearson correlation of 0.81 and an RMSE of 0.18 for sentence-level WER estimation.
- The "black-box" version achieved a Pearson correlation of 0.72 and an RMSE of 0.20.
- The "no-box" e-WER2 system (using acoustic and phonotactic features) achieved a Pearson correlation of 0.56 and an RMSE of 0.24. While lower than the other systems, this is still a significant result for a system with no access to ASR outputs.
- Feature Ablation: The paper shows that combining acoustic and phonotactic features (system F) provides a large improvement in Pearson correlation compared to using acoustic features alone (system E) in the "no-box" setting.
Limitations and Future Work
- Language Specificity: Acknowledged limitation is that the current work is restricted to one language (Arabic). Future work will focus on extending the estimation to different languages and multilingual ASR systems.
- Lattice Rescoring: The authors plan to explore using e-WER for lattice n-best ranking, which could be beneficial for second-pass rescoring in ASR systems.
- Further Exploration of "No-Box" Features: While phonotactic features showed promise, further research into other language-independent features for the "no-box" scenario could yield even better results.
Summary updated