Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions

Chunxi Liu, Michael Picheny, Leda Sari, Pooja Chitkara, Alex Xiao, Xiaohui Zhang, Mark Chou, Andres Alvarado, Caner Hazirbas, Yatharth Saraf

TL;DR

This paper introduces manual transcriptions for the Casual Conversations dataset, an 846-hour corpus designed to audit fairness in AI. By evaluating state-of-the-art Automatic Speech Recognition (ASR) models on this data, the researchers reveal significant performance disparities across gender and skin tone, providing a vital benchmark for building more equitable speech technologies.

What Problem Does This Paper Address?

While algorithmic bias—the tendency for machine learning systems to perform worse for certain demographic groups—has been extensively studied in facial recognition, it remains under-explored in Automatic Speech Recognition (ASR). A major bottleneck has been the lack of large-scale, publicly available speech datasets that include sensitive metadata like age, gender, and skin tone.

Without these labels, researchers cannot easily measure if a model is "fair." For instance, does a voice assistant understand men better than women? Does it struggle with certain dialects or accents correlated with specific ethnicities? This paper addresses this gap by providing high-quality manual transcriptions for a dataset specifically curated for diversity, allowing for a rigorous fairness audit of modern ASR systems.

Methods

The authors evaluated four different ASR models based on the Recurrent Neural Network Transducer (RNN-T) architecture:

  1. LibriSpeech model: Trained on standard "read" speech (audiobooks).
  2. Supervised Video model: Trained on 14,000 hours of manually transcribed social media videos.
  3. Semi-supervised Video model: Trained on over 2 million hours of untranscribed video using pseudo-labeling.
  4. Teacher model: A massive 1-billion parameter model used to generate labels for the semi-supervised process.

To determine if the performance gaps between groups were statistically significant, they employed a Bootstrap Confidence Interval Method. Instead of assuming a normal distribution, they repeatedly resampled the data to see if the Word Error Rate (WER) difference between two subgroups (e.g., Skin Type I vs. Skin Type VI) consistently stayed away from zero.

Datasets

The primary focus is the Casual Conversations dataset:

  • Scope: 45,000 videos (~846 hours) from 3,011 participants.
  • Metadata: Includes self-provided age and gender, and expert-labeled skin tone using the Fitzpatrick scale (Types I to VI).
  • Nature of Speech: Unlike the "read" speech found in datasets like LibriSpeech, this contains "casual" conversational speech with disfluencies (uhs, ums) and colloquialisms.

The models were trained on various combinations of LibriSpeech and massive internal datasets of public, de-identified social media videos.

Key Insights and Ideas

  • The Gender Gap is Real: Across all models, there was a consistent and significant bias. Surprisingly, the models performed better on female speakers than male speakers. This is a notable finding, as historical ASR systems often struggled more with higher-pitched female voices.
  • Skin Tone as a Proxy: While skin tone is a visual trait and shouldn't theoretically affect audio, the researchers found significant WER differences across skin types. They hypothesize that skin tone likely correlates with unobserved variables like dialect or regional accents that do impact ASR performance.
  • Scale Reduces, but Doesn't Erase, Bias: Training on 2 million hours of data (the semi-supervised model) significantly improved overall accuracy compared to the 14k-hour model, but the relative gaps between demographic groups remained stubbornly persistent.
  • Fine-tuning Isn't a Silver Bullet: Even when fine-tuning models on in-domain data from the Casual Conversations set, the relative performance gaps between subgroups did not close. This suggests that unfairness is baked into the models or the training distributions in ways that simple fine-tuning cannot easily fix.

Key Results

The researchers found that the "Video" models (trained on social media data) were much more accurate overall than the LibriSpeech model, yet they exhibited higher relative gaps in fairness.

Table 1: WER results across gender, age, and skin type for various models. Note the bolded relative gaps.

As shown in Figure 4 above, the Video, supervised model had a 37% relative gap between male and female performance. For skin types, the LibriSpeech model showed a massive 37% relative gap between the best and worst-performing skin types.

The statistical significance of these skin-tone gaps is visualized through bootstrap intervals. In the figure below, red lines indicate a statistically significant difference in performance between two skin types.

Figure 1: Confidence intervals for pairwise skin types. Red lines indicate significant differences.

Limitations and Future Work

The authors acknowledge that skin tone is an imperfect variable for ASR; it serves as a proxy for underlying factors like dialect, which are the true drivers of performance disparity. Future work should focus on collecting more explicit labels for dialect and accent.

Additionally, the dataset is currently U.S.-centric. Expanding this work to global populations with a wider variety of languages and regional dialects is essential for a truly comprehensive understanding of ASR fairness. The release of these transcripts is intended as a first step for the community to develop new techniques to mitigate these observed statistical biases.

Summary updated

Abstract

The problem of machine learning systems demonstrating bias towards specific groups of individuals has been studied extensively, particularly in the Facial Recognition area, but much less so in Automatic Speech Recognition (ASR). This paper presents initial Speech Recognition results on “Casual Conversations” – a publicly released 846 hour corpus designed to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of metadata, including age, gender, and skin tone. The entire corpus has been manually transcribed, allowing for detailed ASR evaluations across these metadata. Multiple ASR models are evaluated, including models trained on LibriSpeech, 14,000 hour transcribed, and over 2 million hour untranscribed social media videos. Significant differences in word error rate across gender and skin tone are observed at times for all models. We are releasing human transcripts from the Casual Conversations dataset to encourage the community to develop a variety of techniques to reduce these statistical biases.
BibTeX
@inproceedings{liu2022towards,
  author = {Chunxi Liu and Michael Picheny and Leda Sari and Pooja Chitkara and Alex Xiao and Xiaohui Zhang and Mark Chou and Andres Alvarado and Caner Hazirbas and Yatharth Saraf},
  title = {{Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=KvUdD8j577},
}

Loading related content...

Figure