Abstract
Automatic speech recognition (ASR) systems have been shown to have large quality disparities between the language varieties they are intended or expected to recognize. One way to mitigate this is to train or fine-tune models with more representative datasets. But this approach can be hindered by limited in-domain data for training and evaluation. We propose a new way to improve the robustness of a US English short-form speech recognizer using a small amount of out-of-domain (long-form) African American English (AAE) data. We use CORAAL, YouTube and Mozilla Common Voice to train an audio classifier to approximately output whether an utterance is AAE or some other variety including Mainstream American English (MAE). By combining the classifier output with coarse geographic information, we can select a subset of utterances from a large corpus of untranscribed short-form queries for semi-supervised learning at scale. Fine-tuning on this data results in a 38.5% relative word error rate disparity reduction between AAE and MAE without reducing MAE quality.
The Challenge of Dialectal Disparity
Modern ASR systems—the technology powering voice assistants and transcription tools—often exhibit significantly higher error rates for Black speakers using African American English (AAE) compared to Mainstream American English (MAE). This performance gap is primarily driven by a "data desert": while massive amounts of MAE data are available for training, high-quality, labeled AAE data is scarce. AAE is a linguistically rich variety with distinct phonological and grammatical structures that MAE-centric models often fail to capture. To address this, researchers at Google developed a framework to improve ASR robustness without requiring massive amounts of manually transcribed AAE data.
Methodology: A Three-Stage Pipeline
The proposed solution utilizes a two-stage pipeline to bridge the representation gap through semi-supervised learning.
- AAE Data Selection: The researchers built a binary audio classifier to distinguish AAE from other English varieties. Rather than building from scratch, they used a 600M-parameter Conformer-based foundation model pre-trained on multilingual YouTube data. This was combined with a geographic proxy (targeting regions with high AAE prevalence) to mine a massive corpus of untranscribed short-form voice queries.
- Transcription: These mined utterances were transcribed using a large "teacher" model (a 600M-parameter bidirectional Conformer). This process creates "pseudo-labeled" data suitable for training.
- Fine-tuning: The resulting pseudo-labeled data was used to fine-tune a production-scale 120M-parameter ASR model (a Hybrid Autoregressive Transducer), specifically targeting the AAE variety.
Optimizing the Dialect Classifier
A key technical insight involved identifying the "sweet spot" for fine-tuning the foundation model for dialect detection. The researchers found that updating the middle layers (specifically layers 11 and 12 of the 24-layer model) yielded the highest precision and recall. This suggests that middle layers capture the most relevant phonetic and acoustic features for distinguishing dialects, whereas earlier layers focus on general audio features and later layers on task-specific outputs.
Key Insights and Results
- Disparity Reduction: The best fine-tuned model achieved a 38.5% relative reduction in the Word Error Rate (WER) disparity between AAE and MAE. On acoustic-focused "matched n-gram" tests, the disparity reduction reached 48.7%.
- Quality Over Quantity: Data from the "Southern" dataset (3 million utterances) led to better ASR improvements than the larger "Top 70 Cities" dataset (5.5 million utterances). This indicates that data from regions with a higher density of AAE speakers is more linguistically representative than broad urban sampling.
- Acoustic Gains: By comparing identical phrases spoken in different dialects, the authors proved that the improvements were largely acoustic. The model became significantly better at recognizing the specific sounds and rhythms of AAE.
- Maintaining MAE Performance: Crucially, the fine-tuning process improved AAE recognition without degrading the quality of MAE recognition, ensuring the system remains robust for all users.
Conclusion
This research demonstrates that combining specialized audio classifiers with coarse geographic data is a powerful method for mining representative datasets from unlabeled corpora. By leveraging foundation models and semi-supervised learning, it is possible to significantly reduce performance disparities in ASR systems, moving closer to equitable technology for all speakers of English varieties.
Summary updated