Abstract
Current Automatic Speech Recognition (ASR) auditing practices often overlook the specific needs of marginalized speech communities. This study identifies three major pitfalls in standard audits—narrow standardization, broad demographic categorization, and reliance on Word Error Rate (WER)—and proposes a community-driven framework. Using a case study of speakers with aphasia, the authors demonstrate significant performance disparities and the emergence of "hallucinations" in generative AI models.
Introduction: The Need for Robust Auditing
As ASR technologies move into high-stakes environments like healthcare, job interviews, and legal reporting, robust auditing is paramount. However, current "expert-driven" audits are fundamentally flawed. They typically rely on a single method of text standardization, treat diverse disability groups as monolithic blocks, and use a single metric—Word Error Rate (WER)—that fails to capture the nuances of modern generative AI errors.
This is particularly problematic for speakers with aphasia, a language disorder often caused by stroke or brain injury. These users may rely heavily on ASR to navigate daily life, yet the systems they use are rarely audited with their specific speech patterns or transcription preferences in mind.
Methodology: From Expert-Driven to Community-Driven Audits
The researchers compared a "Standard Audit" approach against a proposed "Community-driven Audit" across six major ASR services: Amazon AWS, AssemblyAI, Google Chirp, Microsoft Azure, OpenAI Whisper, and Rev AI.
Key methodological shifts included:
- Robust Standardization: Testing five variants of text "cleaning" (e.g., removing fillers vs. fragments vs. repetitions) to observe how these choices impact performance rankings.
- Granular Subgroup Analysis: Disaggregating the "aphasia" group into clinical subtypes (fluent vs. non-fluent) and specific classifications (e.g., Broca’s, Anomic).
- Multi-Metric Evaluation: Incorporating metrics like Word Information Lost (WIL) and tracking "hallucination rates" in generative models.
- Community Input: Surveying individuals with aphasia to understand their transcription preferences (e.g., whether they prefer verbatim transcripts or "cleaned" versions reflecting intent).
Data Sources
The study utilized AphasiaBank, a large-scale repository of spontaneous speech interviews. The analysis included 6,185 audio files representing 321 aphasia patients and 216 control speakers. Additionally, a qualitative survey of 7 individuals from the Boston University C.H.A.T. group was conducted to establish community-based ground truth preferences.
Results: Performance Gaps and Standardization Impact
ASR services consistently performed 6-10 percentage points worse for aphasia speakers than for the control group. Microsoft Azure had the highest error rates (17% for aphasia), while Rev AI performed the best (12%).
A critical finding was that the choice of standardization significantly affects the perceived performance of a model. As shown in the evaluation of OpenAI Whisper, different levels of transcript cleaning (from original 'O' to highly refined 'RFFRR') lead to marked differences in measured WER.
Furthermore, treating "aphasia" as a single category hides significant disparities: non-fluent speakers faced a mean WER of 0.21, compared to 0.13 for fluent speakers and 0.07 for the control group.
The Hallucination Hazard
Generative models like OpenAI’s Whisper don't just misspell words; they sometimes "hallucinate" entirely new, plausible-sounding sentences (e.g., adding medical advice or YouTube-style captions) that are not present in the audio. The researchers identified a hallucination rate of 0.9% in Whisper. Crucially, 53 of 56 confirmed instances occurred in the aphasia group, suggesting that dysfluent speech can trigger model fabrications—a unique category of harm that standard WER metrics fail to weight adequately.
Community Preferences and Future Directions
In a notable finding, aphasia speakers actually preferred the most extensively cleaned transcripts. While verbatim transcripts are technically more "accurate" to the audio, they often fail to reflect the speaker's intended message. This suggests that standard audits may be optimizing for the wrong goals by ignoring user preferences.
The authors call for ASR providers to allow users to toggle between "verbatim" and "cleaned" transcription styles to better serve diverse needs. Future work should apply this community-driven framework to other marginalized groups, such as those with accents or other speech impediments like stuttering.
Summary updated