TL;DR
This paper introduces PROPHET, a novel method for prioritizing speech test cases for Automatic Speech Recognition (ASR) systems. By focusing on predicting word-level errors from reference texts alone, PROPHET significantly improves the efficiency of ASR testing and leads to more substantial performance gains when fine-tuning ASR models.
What Problem Does This Paper Address?
Automated Speech Recognition (ASR) systems are increasingly integrated into our daily lives, from virtual assistants to video captioning. However, ensuring their accuracy and robustness is a significant challenge. Testing ASR systems involves collecting and executing speech test cases, each consisting of an audio clip and its corresponding ground-truth text. This process is often expensive and time-consuming. A critical problem is how to strategically select and order these test cases to uncover the most errors with limited resources, whether for initial testing or for fine-tuning models to improve their performance. Existing methods often struggle with the scale of ASR testing, the unavailability of audio data, or lack the granularity needed to pinpoint specific word-level errors.
Methods
The core of PROPHET lies in its ability to predict potential word-level errors in ASR transcriptions without requiring the actual audio or running the ASR system itself. This is achieved through a two-stage process:
- Word-level Error Prediction: PROPHET leverages pre-trained language models like BERT. It tokenizes the reference text into subwords and uses BERT to generate contextualized embeddings for each token. These embeddings are then fed into a shared fully connected network to predict a binary label for each token: whether it's likely to be wrongly recognized by an ASR system. This approach allows for a fine-grained analysis of potential errors at the word level, unlike previous methods that operated at the sentence or phoneme level.
- Test Case Prioritization: Based on the word-level error predictions, PROPHET assigns an "error score" to each test case. This score is the average probability of each subword in the reference text being an error. Test cases with higher error scores are prioritized, as they are more likely to reveal errors in the ASR system. This score is then used to rank test cases, allowing testers to select the most informative ones within a given budget.
Datasets
The study evaluates PROPHET on 12 datasets derived from the L2-Arctic corpus. Each dataset contains approximately 1,130 audio files. For evaluation, 10% of the audio files from each dataset were randomly sampled as test sets. The paper notes that the IndicTTS dataset, mentioned in prior work, was not publicly available, leading to the use of L2-Arctic for replicability.
Key Insights and Ideas
- Text-only prioritization is powerful: PROPHET demonstrates that prioritizing ASR test cases based solely on reference text, without needing audio or running the ASR system, can be highly effective. This is crucial for scenarios where audio is scarce or expensive to obtain.
- Word-level focus matters: Moving beyond sentence or phoneme-level analysis to predict word-level errors significantly improves the ability to identify problematic test cases.
- Pre-trained models are game-changers: Leveraging large language models like BERT allows for effective error prediction even with limited labeled training data for the predictor itself.
- Error density is a better predictor than diversity: The study found that the original model's Word Error Rate (WER) on a test set is a stronger indicator of that set's potential to improve ASR performance than metrics related to phoneme or triphone diversity. This challenges some prior assumptions in ASR testing.
- Diversity can sometimes hurt: Considering phoneme/triphone diversity in test case selection, as done by some baseline methods, was found to sometimes harm the effectiveness of error prediction and prioritization.
Key Results
- Uncovering Errors: PROPHET uncovered 12.63% more wrongly recognized words than the second-best method on average, across 3 ASR systems and 12 datasets. It outperformed baseline methods in 35 out of 36 configurations.
- Improving ASR Performance: When test cases prioritized by PROPHET were used to fine-tune ASR systems, the resulting performance improvements were significantly larger than those achieved by fine-tuning with test cases selected by other methods. Statistical tests confirmed PROPHET's superiority.
- Correlation Analysis: The original model's WER on a fine-tuning set showed a stronger positive correlation with performance improvement than phoneme diversity metrics.
Limitations and Future Work
- Generalizability: While the study used 3 ASR systems and 12 datasets with varying accents, the authors acknowledge that the results might not be universally generalizable to all models and datasets. They encourage further research to extend the study.
- Synthetic Audio: The paper suggests exploring the value of using synthetic audio to improve ASR systems as a future direction.
- Larger Scale: Extending the study to more ASR systems and larger datasets is also proposed for future work.
Summary updated