Error-Driven Fixed-Budget ASR Personalization for Accented Speakers

Abhijeet Awasthi, Aman Kansal, Sunita Sarawagi, Preethi Jyothi

TL;DR

This paper tackles the challenge of personalizing Automatic Speech Recognition (ASR) models for speakers with accents, a task often hindered by limited data collection budgets. They introduce an "error-driven" sentence selection method that intelligently picks sentences likely to be misrecognized by the ASR model, leading to more effective personalization with less data compared to random selection.

What Problem Does This Paper Address?

Modern ASR systems perform remarkably well for native speakers, but their accuracy plummets when encountering accented speech. This disparity creates an accessibility gap, especially as voice interfaces become ubiquitous. While fine-tuning ASR models with speaker-specific data is a known solution, collecting this data is costly and time-consuming. The core problem this paper addresses is how to efficiently select a small, fixed set of sentences for a speaker to record, such that fine-tuning an ASR model on these recordings yields the greatest improvement in accuracy for that speaker's accent. Existing methods for sentence selection are often limited to phonetic diversity, which doesn't directly account for the ASR model's specific weaknesses for a given accent.

Methods

The authors propose an "error-driven" approach to sentence selection. The central idea is to identify sentences that the current ASR model is likely to struggle with for a specific speaker. To achieve this, they first train a "phoneme-level error detector" using a small, initial set of speaker-specific utterances (a "seed set"). This detector learns to predict the probability of a phoneme being misrecognized.

When selecting sentences from a larger corpus for recording, their algorithm combines two factors:

  1. Predicted Error Probability: Sentences with phonemes that the error detector predicts are likely to be misrecognized are favored.
  2. Phonetic Balance: To avoid over-selecting sentences that only target a few types of errors, a penalty is applied for over-represented phonemes in the already selected set. This encourages a more balanced coverage of phonetic challenges.

The final score for a sentence is a combination of these two terms, normalized by sentence length to prevent bias towards longer utterances. The algorithm iteratively selects sentences until the budget (number of sentences to record) is met.

Datasets

The experiments were conducted on two datasets:

  • IndicTTS: This dataset comprises English utterances from speakers with diverse Indian accents (Kannada, Malayalam, Rajasthani, Assamese, Gujarati, Hindi, Manipuri, Tamil). The corpus size (for selection) was around 4.3K sentences, with test sets of 1.9K sentences.
  • L2-Arctic: This dataset uses sentences from the CMU-Arctic dataset but features non-native English speakers. Speakers included Vietnamese, Mandarin, Spanish, Korean, and Arabic native speakers. The corpus size was 690 sentences, with test sets of 340 sentences.

For training the error detector, a small seed set (50 sentences) and a development set (50 sentences) were used from each speaker's data.

Key Insights and Ideas

  • Focusing on Weaknesses is Key: The most impactful insight is that personalizing an ASR model is most effective when focusing on the specific errors the model makes for a given speaker, rather than just general phonetic diversity.
  • Tiny Seed Sets are Powerful: A surprisingly small amount of speaker-specific data (the seed set) is sufficient to train a phoneme-level error detector that accurately predicts where the ASR model will struggle.
  • Balancing Errors and Diversity: Simply picking the "hardest" sentences might lead to a narrow set of errors. The proposed method's inclusion of a phonetic balance term is crucial for achieving broader improvements.
  • Efficiency Matters: The paper demonstrates that their method significantly reduces the amount of data needed to achieve a certain level of ASR improvement, directly addressing the budget constraint.

Key Results

  • Significant WER Reduction: Fine-tuning with sentences selected by their error-driven method consistently yielded lower Word Error Rates (WER) compared to random selection, even with the same number of recorded utterances.
  • Improved Efficiency: For example, on the IndicTTS dataset, their method achieved a similar WER reduction using 100 sentences as random selection did with 250 sentences for one speaker, and 135 sentences versus 250 for another.
  • Error Model Effectiveness: Utterances from sentences selected by their error model showed significantly higher WERs on the pretrained ASR model compared to utterances from randomly selected sentences, validating the error detector's ability to identify challenging inputs.
  • Phonetic Diversity Baseline Underperforms: The "Diverse" selection strategy (based on phonetic richness) did not show significant improvements over random selection, highlighting the advantage of the error-driven approach.

Limitations and Future Work

  • Limitations: The paper acknowledges that the "Diverse" selection method (based on phonetic richness) did not outperform random selection, suggesting that phonetic diversity alone is not sufficient. The improvement on L2-Arctic was smaller than IndicTTS, potentially due to the smaller selection corpus size.
  • Future Work: The authors express interest in applying their method to personalize ASR for dysarthric speech, which presents a different set of challenges related to speech intelligibility.

Summary updated

Abstract

We consider the task of personalizing ASR models while being constrained by a fixed budget on recording speaker specific utterances. Given a speaker and an ASR model, we propose a method of identifying sentences for which the speaker’s utterances are likely to be harder for the given ASR model to recognize. We assume a tiny amount of speaker-specific data to learn phoneme-level error models which help us select such sentences. We show that speaker’s utterances on the sentences selected using our error model indeed have larger error rates when compared to speaker’s utterances on randomly selected sentences. We find that fine-tuning the ASR model on the sentence utterances selected with the help of error models yield higher WER improvements in comparison to fine-tuning on an equal number of randomly selected sentence utterances. Thus, our method provides an efficient way of collecting speaker utterances under budget constraints for personalizing ASR models.
BibTeX
@inproceedings{awasthi2021error,
  author = {Abhijeet Awasthi and Aman Kansal and Sunita Sarawagi and Preethi Jyothi},
  title = {{Error-Driven Fixed-Budget ASR Personalization for Accented Speakers}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=9FmkUDRIz0A},
}

Loading related content...

Figure