Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions

Chunxi Liu, Michael Picheny, Leda Sari, Pooja Chitkara, Alex Xiao, Xiaohui Zhang, Mark Chou, Andres Alvarado, Caner Hazirbas, Yatharth Saraf

Abstract: It is well known that many machine learning systems demonstrate bias towards specific groups of individuals. This problem has been studied extensively in the Facial Recognition area, but much less so in Automatic Speech Recognition (ASR). This paper presents initial Speech Recognition results on "Casual Conversations" -- a publicly released 846 hour corpus designed to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of metadata, including age, gender, and skin tone. The entire corpus has been manually transcribed, allowing for detailed ASR evaluations across these metadata. Multiple ASR models are evaluated, including models trained on LibriSpeech, 14,000 hour transcribed, and over 2 million hour untranscribed social media videos. Significant differences in word error rate across gender and skin tone are observed at times for all models. We are releasing human transcripts from the Casual Conversations dataset to encourage the community to develop a variety of techniques to reduce these statistical biases.

Introduction

Algorithmic bias—the tendency for machine learning systems to perform worse for certain demographic groups—has been extensively studied in facial recognition, but remains under-explored in Automatic Speech Recognition (ASR). A major bottleneck has been the lack of large-scale, publicly available speech datasets that include sensitive metadata like age, gender, and skin tone. Without these labels, researchers cannot easily measure if a model is "fair." This paper addresses this gap by providing high-quality manual transcriptions for the Casual Conversations dataset, a corpus specifically curated for diversity, allowing for a rigorous fairness audit of modern ASR systems.

The Casual Conversations Dataset

The Casual Conversations dataset consists of approximately 45,000 videos (~846 hours) featuring 3,011 participants. Crucially, it includes self-provided age and gender metadata, along with expert-labeled skin tone using the Fitzpatrick scale (Types I to VI). Unlike the "read" speech found in datasets like LibriSpeech, this corpus contains "casual" conversational speech characterized by disfluencies (uhs, ums) and colloquialisms, providing a more realistic testbed for ASR performance.

Methodology

The authors evaluated four ASR models based on the Recurrent Neural Network Transducer (RNN-T) architecture:

  1. LibriSpeech model: Trained on standard read speech (audiobooks).
  2. Supervised Video model: Trained on 14,000 hours of manually transcribed social media videos.
  3. Semi-supervised Video model: Trained on over 2 million hours of untranscribed video using pseudo-labeling.
  4. Teacher model: A massive 1-billion parameter model used for generating semi-supervised labels.

To assess statistical significance, a Bootstrap Confidence Interval Method was employed. By repeatedly resampling the data, the researchers determined if Word Error Rate (WER) differences between subgroups consistently deviated from zero, indicating a non-random performance gap.

Results and Analysis

The evaluation revealed significant performance disparities across demographic groups. Notably, models generally performed better on female speakers than male speakers, a shift from historical ASR trends where higher-pitched voices were often more challenging.

Table 1: WER results across gender, age, and skin type for various models. Note the bolded relative gaps.

As shown in Table 1, the Video models trained on social media data achieved higher overall accuracy than the LibriSpeech model but exhibited larger relative fairness gaps. For example, the supervised video model showed a 37% relative gap between male and female performance. For skin types, the LibriSpeech model showed a massive 37% relative gap between the best and worst-performing skin types.

Significant WER differences were also observed across skin types. While skin tone is a visual trait, it likely serves as a proxy for unobserved variables such as dialect or regional accents. The statistical significance of these gaps is visualized in Figure 1.

Figure 1: Confidence intervals for pairwise skin types. Red lines indicate significant differences.

Key findings include:

  • Scale vs. Bias: Increasing training data to 2 million hours improved overall accuracy but did not eliminate relative performance gaps.
  • Fine-tuning Limitations: Fine-tuning on in-domain data failed to close the relative gaps between subgroups, suggesting that bias may be deeply embedded in model architectures or training distributions.

Conclusion

This work highlights the persistence of demographic bias in state-of-the-art ASR systems. While skin tone is an imperfect variable, its correlation with performance gaps underscores the need for more explicit dialect and accent labeling. The release of the Casual Conversations transcripts is intended to catalyze community efforts in developing robust techniques to mitigate statistical biases and build more equitable speech technologies.

Summary updated

Abstract

It is well known that many machine learning systems demonstrate bias towards specific groups of individuals. This problem has been studied extensively in the Facial Recognition area, but much less so in Automatic Speech Recognition (ASR). This paper presents initial Speech Recognition results on "Casual Conversations" -- a publicly released 846 hour corpus designed to help researchers evaluate their computer vision and audio models for accuracy across a diverse set of metadata, including age, gender, and skin tone. The entire corpus has been manually transcribed, allowing for detailed ASR evaluations across these metadata. Multiple ASR models are evaluated, including models trained on LibriSpeech, 14,000 hour transcribed, and over 2 million hour untranscribed social media videos. Significant differences in word error rate across gender and skin tone are observed at times for all models. We are releasing human transcripts from the Casual Conversations dataset to encourage the community to develop a variety of techniques to reduce these statistical biases.
BibTeX
@inproceedings{liu2022towards,
  author = {Chunxi Liu and Michael Picheny and Leda Sari and Pooja Chitkara and Alex Xiao and Xiaohui Zhang and Mark Chou and Andres Alvarado and Caner Hazirbas and Yatharth Saraf},
  title = {{Towards Measuring Fairness in Speech Recognition: Casual Conversations Dataset Transcriptions}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=KvUdD8j577},
}
Figure