Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia

Katelyn Xiaoying Mei, Anna Seo Gyeong Choi, Hilke Schellmann, Mona Sloane, Allison Koenecke

Abstract

Current Automatic Speech Recognition (ASR) auditing practices often overlook the specific needs of marginalized speech communities. This study identifies three major pitfalls in standard audits—narrow standardization, broad demographic categorization, and reliance on Word Error Rate (WER)—and proposes a community-driven framework. Using a case study of speakers with aphasia, the authors demonstrate significant performance disparities and the emergence of "hallucinations" in generative AI models.

Introduction: The Need for Robust Auditing

As ASR technologies move into high-stakes environments like healthcare, job interviews, and legal reporting, robust auditing is paramount. However, current "expert-driven" audits are fundamentally flawed. They typically rely on a single method of text standardization, treat diverse disability groups as monolithic blocks, and use a single metric—Word Error Rate (WER)—that fails to capture the nuances of modern generative AI errors.

This is particularly problematic for speakers with aphasia, a language disorder often caused by stroke or brain injury. These users may rely heavily on ASR to navigate daily life, yet the systems they use are rarely audited with their specific speech patterns or transcription preferences in mind.

Methodology: From Expert-Driven to Community-Driven Audits

The researchers compared a "Standard Audit" approach against a proposed "Community-driven Audit" across six major ASR services: Amazon AWS, AssemblyAI, Google Chirp, Microsoft Azure, OpenAI Whisper, and Rev AI.

Comparison of the traditional Expert-driven Standard Audit versus the proposed Community-driven Audit framework.

Key methodological shifts included:

  • Robust Standardization: Testing five variants of text "cleaning" (e.g., removing fillers vs. fragments vs. repetitions) to observe how these choices impact performance rankings.
  • Granular Subgroup Analysis: Disaggregating the "aphasia" group into clinical subtypes (fluent vs. non-fluent) and specific classifications (e.g., Broca’s, Anomic).
  • Multi-Metric Evaluation: Incorporating metrics like Word Information Lost (WIL) and tracking "hallucination rates" in generative models.
  • Community Input: Surveying individuals with aphasia to understand their transcription preferences (e.g., whether they prefer verbatim transcripts or "cleaned" versions reflecting intent).

Data Sources

The study utilized AphasiaBank, a large-scale repository of spontaneous speech interviews. The analysis included 6,185 audio files representing 321 aphasia patients and 216 control speakers. Additionally, a qualitative survey of 7 individuals from the Boston University C.H.A.T. group was conducted to establish community-based ground truth preferences.

Results: Performance Gaps and Standardization Impact

ASR services consistently performed 6-10 percentage points worse for aphasia speakers than for the control group. Microsoft Azure had the highest error rates (17% for aphasia), while Rev AI performed the best (12%).

A critical finding was that the choice of standardization significantly affects the perceived performance of a model. As shown in the evaluation of OpenAI Whisper, different levels of transcript cleaning (from original 'O' to highly refined 'RFFRR') lead to marked differences in measured WER.

Word Error Rate (WER) for OpenAI Whisper across different standardization levels, showing how 'cleaning' choices impact measured performance.

Furthermore, treating "aphasia" as a single category hides significant disparities: non-fluent speakers faced a mean WER of 0.21, compared to 0.13 for fluent speakers and 0.07 for the control group.

The Hallucination Hazard

Generative models like OpenAI’s Whisper don't just misspell words; they sometimes "hallucinate" entirely new, plausible-sounding sentences (e.g., adding medical advice or YouTube-style captions) that are not present in the audio. The researchers identified a hallucination rate of 0.9% in Whisper. Crucially, 53 of 56 confirmed instances occurred in the aphasia group, suggesting that dysfluent speech can trigger model fabrications—a unique category of harm that standard WER metrics fail to weight adequately.

Community Preferences and Future Directions

In a notable finding, aphasia speakers actually preferred the most extensively cleaned transcripts. While verbatim transcripts are technically more "accurate" to the audio, they often fail to reflect the speaker's intended message. This suggests that standard audits may be optimizing for the wrong goals by ignoring user preferences.

The authors call for ASR providers to allow users to toggle between "verbatim" and "cleaned" transcription styles to better serve diverse needs. Future work should apply this community-driven framework to other marginalized groups, such as those with accents or other speech impediments like stuttering.

Summary updated

Abstract

Automatic Speech Recognition (ASR) systems' growing use warrants robust auditing approaches to ensure equitable transcription quality, especially for people with speech disorders like aphasia who disproportionately depend on ASR. While academic and industry audits have revealed performance disparities across user populations, standard auditing practices often overlook nuances that risk masking harm to marginalized groups. We identify three common pitfalls in standard ASR audits: (1) adhering to one method of text standardization, which can mask variance in ASR performance and ignore the standardization preferences of marginalized communities; (2) displaying high-level demographic findings without considering performance disparities by nuanced intersectional subgroups, or conditioning on relevant acoustic properties; and (3) reporting only one gold-standard metric (Word Error Rate), which inadequately quantifies common generative AI errors like hallucinations. We propose a holistic auditing framework addressing these pitfalls, and in a case study of six popular ASR systems, find consistently worse ASR performance for speakers with aphasia relative to a control group. We call on practitioners to implement these robust, community-driven ASR auditing practices better suited for the rapidly changing ASR landscape.
BibTeX
@inproceedings{mei2026addressing,
  author = {Katelyn Xiaoying Mei and Anna Seo Gyeong Choi and Hilke Schellmann and Mona Sloane and Allison Koenecke},
  title = {{Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasia}},
  booktitle = {FAccT},
  year = {2026},
  doi = {10.1145/3805689.3812320},
}
Figure