Performance Disparities between Accents in Automatic Speech Recognition (Student Abstract)

Alex DiChristofano, Henry Shuster, Shefali Chandra, Neal PatwariAAAI 2023 · DOI

TL;DR

This paper reveals that popular Automatic Speech Recognition (ASR) services exhibit significant performance disparities based on a speaker's country of origin, even when controlling for linguistic factors. Specifically, ASR accuracy is correlated with a speaker's birth country's geopolitical alignment with the United States, suggesting a bias beyond purely linguistic differences.

What Problem Does This Paper Address?

The increasing ubiquity of Automatic Speech Recognition (ASR) systems in our daily lives, from voice assistants to transcription services, hinges on their ability to accurately understand diverse human speech. However, a growing body of research highlights that ASR systems often perform worse for certain demographic groups, particularly those with non-standard accents. This paper delves into this issue by investigating whether geopolitical factors, beyond linguistic ones, contribute to performance disparities in widely used ASR services. The authors aim to understand if ASR systems are implicitly biased towards speech patterns from countries that have closer political ties to the United States.

Methods

The researchers conducted a large-scale audit of three top cloud-based ASR services: Microsoft, Amazon, and Google. They utilized data from the Speech Accent Archive, which contains recordings of speakers from around the world reading a standardized passage designed to capture a wide range of English sounds and clusters.

For their analysis, they collected speaker information including age, sex, country of birth, first language, age of onset of English speaking, and English learning environment. They then performed a group-level analysis comparing Word Information Lost (WIL) – a metric where lower is better, indicating higher accuracy – across different first languages and ASR services.

To further dissect the factors influencing performance, they employed a linear regression model. This model aimed to quantify the impact of various speaker-level covariates, including age, sex, English learning environment, whether the first language was Germanic (a proxy for linguistic similarity to English), and crucially, whether the speaker's birth country was a member of NATO (as a proxy for geopolitical alignment with the US).

Datasets

The primary dataset used in this study is the Speech Accent Archive. This archive contains recordings of speakers from diverse linguistic backgrounds reading a standardized passage. The authors state that their dataset includes 2,713 speakers, representing 212 first languages across 171 birth countries. They also highlight the top ten first languages represented in their data: English, Spanish, Arabic, Mandarin, Korean, Russian, French, Portuguese, Dutch, and German.

Key Insights and Ideas

  • Geopolitical Bias in ASR: The most striking finding is the statistically significant correlation between a speaker's birth country's geopolitical alignment with the United States (measured by NATO membership) and ASR performance. This suggests that ASR systems may be implicitly biased towards speech patterns from countries that are politically closer to the US, even when controlling for linguistic factors.
  • Beyond Linguistic Similarity: While it's expected that ASR systems perform better for speakers whose first language is English, this study demonstrates that geopolitical factors play a role in addition to linguistic similarity. The performance gap persists even when controlling for the influence of English as a first language.
  • Impact of Learning Environment: Speakers who learned English in a naturalistic environment showed significantly lower WIL (better performance) compared to those who learned in an academic setting.
  • Age of Onset Matters: A later age of onset for English speaking is correlated with increased WIL, indicating poorer ASR performance.
  • "United States English" Setting is Not a Panacea: Even when using the most accurate transcription setting available for Google (which showed the largest disparities), a significant performance gap remained, suggesting that current efforts to offer diverse transcription options may not fully address these underlying biases.

Key Results

  • All evaluated ASR services (Microsoft, Amazon, Google) performed significantly better for speakers whose first language was English. On average, WIL was 0.14 lower for these speakers.
  • Across all services, Microsoft performed best, followed by Amazon, and then Google.
  • The worst performance was observed for speakers whose first language was Mandarin, Spanish, or Russian.
  • The regression analysis revealed that being born in a NATO country was associated with a significantly lower WIL, even after controlling for other factors like being born in the United States. This suggests a tangible performance advantage for speakers from countries aligned with US geopolitical power.
  • WIL significantly increases with a later age of onset of English speaking.

Limitations and Future Work

  • Proxy for Geopolitical Power: The study uses NATO membership as a proxy for geopolitical alignment with the United States. While a reasonable simplification, it's acknowledged that this is not a perfect measure and a more nuanced approach to quantifying geopolitical influence could be explored.
  • "United States English" Setting: The default setting for most ASR services tested was "United States English." While the authors did explore "Google All Settings" to find the most accurate transcription, they note that significant disparities persisted. Future work could involve a more exhaustive analysis of all available language settings across all services.
  • Limited Scope of "Accent": The study focuses on performance disparities related to country of origin and first language. While these are significant factors, future research could delve deeper into other aspects of accent variation and their impact on ASR.
  • Future Work: The authors suggest that current efforts to offer different global transcription options do not fully mitigate performance disparities. Future work should focus on developing ASR systems that are more equitable across diverse linguistic backgrounds, potentially by addressing the underlying biases in training data or model architectures.

Summary updated

BibTeX
@inproceedings{dichristofano2023performance,
  author = {Alex DiChristofano and Henry Shuster and Shefali Chandra and Neal Patwari},
  title = {{Performance Disparities between Accents in Automatic Speech Recognition (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.26960},
}
Figure