Prioritizing Speech Test Cases

Zhou Yang, Jieke Shi, Muhammad Hilmi Asyrofi, Bowen Xu, Xin Zhou, Dong-Gyun Han, David Lo

TL;DR

This paper introduces PROPHET, a novel method for prioritizing speech test cases for Automatic Speech Recognition (ASR) systems. By focusing on predicting word-level errors from reference texts alone, PROPHET significantly improves the efficiency of ASR testing and leads to more substantial performance gains when fine-tuning ASR models.

What Problem Does This Paper Address?

Automated Speech Recognition (ASR) systems are increasingly integrated into our daily lives, from virtual assistants to video captioning. However, ensuring their accuracy and robustness is a significant challenge. Testing ASR systems involves collecting and executing speech test cases, each consisting of an audio clip and its corresponding ground-truth text. This process is often expensive and time-consuming. A critical problem is how to strategically select and order these test cases to uncover the most errors with limited resources, whether for initial testing or for fine-tuning models to improve their performance. Existing methods often struggle with the scale of ASR testing, the unavailability of audio data, or lack the granularity needed to pinpoint specific word-level errors.

Methods

The core of PROPHET lies in its ability to predict potential word-level errors in ASR transcriptions without requiring the actual audio or running the ASR system itself. This is achieved through a two-stage process:

  1. Word-level Error Prediction: PROPHET leverages pre-trained language models like BERT. It tokenizes the reference text into subwords and uses BERT to generate contextualized embeddings for each token. These embeddings are then fed into a shared fully connected network to predict a binary label for each token: whether it's likely to be wrongly recognized by an ASR system. This approach allows for a fine-grained analysis of potential errors at the word level, unlike previous methods that operated at the sentence or phoneme level.
  2. Test Case Prioritization: Based on the word-level error predictions, PROPHET assigns an "error score" to each test case. This score is the average probability of each subword in the reference text being an error. Test cases with higher error scores are prioritized, as they are more likely to reveal errors in the ASR system. This score is then used to rank test cases, allowing testers to select the most informative ones within a given budget.

Datasets

The study evaluates PROPHET on 12 datasets derived from the L2-Arctic corpus. Each dataset contains approximately 1,130 audio files. For evaluation, 10% of the audio files from each dataset were randomly sampled as test sets. The paper notes that the IndicTTS dataset, mentioned in prior work, was not publicly available, leading to the use of L2-Arctic for replicability.

Key Insights and Ideas

  • Text-only prioritization is powerful: PROPHET demonstrates that prioritizing ASR test cases based solely on reference text, without needing audio or running the ASR system, can be highly effective. This is crucial for scenarios where audio is scarce or expensive to obtain.
  • Word-level focus matters: Moving beyond sentence or phoneme-level analysis to predict word-level errors significantly improves the ability to identify problematic test cases.
  • Pre-trained models are game-changers: Leveraging large language models like BERT allows for effective error prediction even with limited labeled training data for the predictor itself.
  • Error density is a better predictor than diversity: The study found that the original model's Word Error Rate (WER) on a test set is a stronger indicator of that set's potential to improve ASR performance than metrics related to phoneme or triphone diversity. This challenges some prior assumptions in ASR testing.
  • Diversity can sometimes hurt: Considering phoneme/triphone diversity in test case selection, as done by some baseline methods, was found to sometimes harm the effectiveness of error prediction and prioritization.

Key Results

  • Uncovering Errors: PROPHET uncovered 12.63% more wrongly recognized words than the second-best method on average, across 3 ASR systems and 12 datasets. It outperformed baseline methods in 35 out of 36 configurations.
  • Improving ASR Performance: When test cases prioritized by PROPHET were used to fine-tune ASR systems, the resulting performance improvements were significantly larger than those achieved by fine-tuning with test cases selected by other methods. Statistical tests confirmed PROPHET's superiority.
  • Correlation Analysis: The original model's WER on a fine-tuning set showed a stronger positive correlation with performance improvement than phoneme diversity metrics.

Limitations and Future Work

  • Generalizability: While the study used 3 ASR systems and 12 datasets with varying accents, the authors acknowledge that the results might not be universally generalizable to all models and datasets. They encourage further research to extend the study.
  • Synthetic Audio: The paper suggests exploring the value of using synthetic audio to improve ASR systems as a future direction.
  • Larger Scale: Extending the study to more ASR systems and larger datasets is also proposed for future work.

Summary updated

Abstract

With the wide adoption of automated speech recognition (ASR) systems, it is increasingly important to test and improve ASR systems. However, collecting and executing speech test cases is usually expensive and time-consuming, motivating us to strategically prioritize speech test cases. A key question is: how to determine the ideal order of collecting and executing speech test cases to uncover more errors as early as possible? Each speech test case consists of a piece of audio and the corresponding reference text. In this work, we propose PROPHET (PRiOritizing sPeecH tEsT), a tool that predicts potential error-uncovering speech test cases only based on their reference texts. Thus, PROPHET analyzes test cases and prioritizes them without running the ASR system, which can analyze speech test cases at a large scale. We evaluate 6 different prioritization methods on 3 ASR systems and 12 datasets. Given the same testing budget, we find that our approach uncovers 12.63% more wrongly recognized words than the state-of-the-art method. We select test cases from the prioritized list to fine-tune ASR systems and analyze how our approach can improve the ASR system performance. Statistical tests show that our proposed method can bring significantly larger performance improvement to ASR systems than the existing baseline methods. Furthermore, we perform correlation analysis and confirm that fine-tuning an ASR system using a dataset, on which the model performs worse, tends to improve the performance more.
BibTeX
@article{yang2023prioritizing,
  author = {Zhou Yang and Jieke Shi and Muhammad Hilmi Asyrofi and Bowen Xu and Xin Zhou and Dong-Gyun Han and David Lo},
  title = {{Prioritizing Speech Test Cases}},
  journal = {ACM Trans. Softw. Eng. Methodol.},
  year = {2023},
}
Figure