TL;DR
This paper introduces manual transcriptions for the Casual Conversations dataset, an 846-hour corpus designed to audit fairness in AI. By evaluating state-of-the-art Automatic Speech Recognition (ASR) models on this data, the researchers reveal significant performance disparities across gender and skin tone, providing a vital benchmark for building more equitable speech technologies.
What Problem Does This Paper Address?
While algorithmic bias—the tendency for machine learning systems to perform worse for certain demographic groups—has been extensively studied in facial recognition, it remains under-explored in Automatic Speech Recognition (ASR). A major bottleneck has been the lack of large-scale, publicly available speech datasets that include sensitive metadata like age, gender, and skin tone.
Without these labels, researchers cannot easily measure if a model is "fair." For instance, does a voice assistant understand men better than women? Does it struggle with certain dialects or accents correlated with specific ethnicities? This paper addresses this gap by providing high-quality manual transcriptions for a dataset specifically curated for diversity, allowing for a rigorous fairness audit of modern ASR systems.
Methods
The authors evaluated four different ASR models based on the Recurrent Neural Network Transducer (RNN-T) architecture:
- LibriSpeech model: Trained on standard "read" speech (audiobooks).
- Supervised Video model: Trained on 14,000 hours of manually transcribed social media videos.
- Semi-supervised Video model: Trained on over 2 million hours of untranscribed video using pseudo-labeling.
- Teacher model: A massive 1-billion parameter model used to generate labels for the semi-supervised process.
To determine if the performance gaps between groups were statistically significant, they employed a Bootstrap Confidence Interval Method. Instead of assuming a normal distribution, they repeatedly resampled the data to see if the Word Error Rate (WER) difference between two subgroups (e.g., Skin Type I vs. Skin Type VI) consistently stayed away from zero.
Datasets
The primary focus is the Casual Conversations dataset:
- Scope: 45,000 videos (~846 hours) from 3,011 participants.
- Metadata: Includes self-provided age and gender, and expert-labeled skin tone using the Fitzpatrick scale (Types I to VI).
- Nature of Speech: Unlike the "read" speech found in datasets like LibriSpeech, this contains "casual" conversational speech with disfluencies (uhs, ums) and colloquialisms.
The models were trained on various combinations of LibriSpeech and massive internal datasets of public, de-identified social media videos.
Key Insights and Ideas
- The Gender Gap is Real: Across all models, there was a consistent and significant bias. Surprisingly, the models performed better on female speakers than male speakers. This is a notable finding, as historical ASR systems often struggled more with higher-pitched female voices.
- Skin Tone as a Proxy: While skin tone is a visual trait and shouldn't theoretically affect audio, the researchers found significant WER differences across skin types. They hypothesize that skin tone likely correlates with unobserved variables like dialect or regional accents that do impact ASR performance.
- Scale Reduces, but Doesn't Erase, Bias: Training on 2 million hours of data (the semi-supervised model) significantly improved overall accuracy compared to the 14k-hour model, but the relative gaps between demographic groups remained stubbornly persistent.
- Fine-tuning Isn't a Silver Bullet: Even when fine-tuning models on in-domain data from the Casual Conversations set, the relative performance gaps between subgroups did not close. This suggests that unfairness is baked into the models or the training distributions in ways that simple fine-tuning cannot easily fix.
Key Results
The researchers found that the "Video" models (trained on social media data) were much more accurate overall than the LibriSpeech model, yet they exhibited higher relative gaps in fairness.
As shown in Figure 4 above, the Video, supervised model had a 37% relative gap between male and female performance. For skin types, the LibriSpeech model showed a massive 37% relative gap between the best and worst-performing skin types.
The statistical significance of these skin-tone gaps is visualized through bootstrap intervals. In the figure below, red lines indicate a statistically significant difference in performance between two skin types.
Limitations and Future Work
The authors acknowledge that skin tone is an imperfect variable for ASR; it serves as a proxy for underlying factors like dialect, which are the true drivers of performance disparity. Future work should focus on collecting more explicit labels for dialect and accent.
Additionally, the dataset is currently U.S.-centric. Expanding this work to global populations with a wider variety of languages and regional dialects is essential for a truly comprehensive understanding of ASR fairness. The release of these transcripts is intended as a first step for the community to develop new techniques to mitigate these observed statistical biases.
Summary updated