TL;DR
This paper demonstrates that by incorporating a small fraction of disordered speech data into the training of a large, general-purpose Automatic Speech Recognition (ASR) model, we can significantly improve its performance on disordered speech without degrading performance on typical speech. This approach offers a more accessible and equitable solution than relying solely on individual user personalization.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are becoming ubiquitous, but they often struggle with disordered speech, leading to suboptimal performance for individuals with conditions like Parkinson's, ALS, or Down Syndrome. While personalized ASR models can improve accuracy for individual users, this approach has limitations: it requires users to provide their own speech data, which can be difficult or impossible for some, and it incurs maintenance costs. The paper addresses the challenge of creating a single, general-purpose ASR model that can effectively recognize disordered speech "out of the box," making speech technology more accessible to a wider range of users.
Methods
The researchers leverage Google's Universal Speech Model (USM), a large, multilingual Conformer-based ASR model. They fine-tune this pre-trained model by incorporating a dataset of disordered speech recordings. The core of their method involves experimenting with different "training weights" for this disordered speech data, essentially controlling its proportion within the overall fine-tuning dataset. They analyze how these varying weights impact performance on both disordered and typical speech benchmarks, aiming to find an optimal balance.
Datasets
- Prompted Speech Dataset: Derived from the Euphonia corpus, this dataset consists of approximately 1,000 hours of disordered speech samples where participants repeat pre-defined phrases. It includes rich metadata on speaker demographics, etiology, and severity of speech disorder.
- Conversational Speech Test Set: A newly compiled test set of spontaneous, unprompted disordered speech. This set is more challenging as it reflects natural conversational patterns, including self-repairs and varied speaking rates.
- Typical Speech Benchmarks: Standard ASR evaluation sets, including a multilingual test set (English and 18 other languages) and the Librispeech dataset, were used to assess performance on typical speech.
Key Insights and Ideas
- Small Data, Big Impact: Surprisingly, even a very small amount of disordered speech data (less than 1% of the fine-tuning data) can lead to substantial improvements in recognition accuracy for disordered speech.
- Win-Win Scenario: The proposed fine-tuning strategy improves disordered speech recognition without negatively impacting performance on standard, typical speech benchmarks.
- Bridging the Gap: The approach significantly narrows the performance gap between a general-purpose ASR model and highly personalized models for disordered speech.
- No Special Labeling Needed: The disordered speech data was integrated into the training set without any special labels or signals to distinguish it from typical speech, suggesting that long-tail speech patterns can be incorporated early in the training process.
Key Results
- Prompted Speech: A 33% improvement in Word Error Rate (WER) was observed on the prompted speech dataset when using an optimal training weight for the disordered speech data.
- Conversational Speech: A 26% improvement in WER was achieved on the newly gathered spontaneous, conversational disordered speech dataset.
- Typical Speech: Performance on typical speech benchmarks (multilingual test set and Librispeech) remained largely unaffected, with only minor degradation at the highest training weights.
- Personalization Gap: The fine-tuned USM model closed 64% of the performance gap between the baseline USM and a personalized RNN-T model.
- Model Architecture: The USM architecture itself provided a strong baseline, and the addition of disordered speech data further enhanced its capabilities.
Limitations and Future Work
- Data Availability: The Euphonia corpus datasets used in this study cannot be made publicly available due to participant consent. However, the Speech Accessibility Project is mentioned as an alternative source for disordered speech data.
- Optimal Mixture Weights: Investigating the ideal proportions of disordered and typical speech data in the training mix is an interesting future direction.
- Multilingual Expansion: Acquiring and incorporating disordered speech data in languages other than English would further enhance the model's global accessibility.
- Pseudo-labeling: The generalized model could potentially be used for pseudo-labeling more disordered speech recordings, which can then be used to further improve speaker-independent models.
- Personalization Still Relevant: For individuals with severe speech impairments or those requiring extremely high accuracy in specific domains, personalized models may still be the best solution. However, the proposed approach offers a strong baseline for many users.
Summary updated