Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech

Dena F. Mujtaba, Nihar R. Mahapatra, Megan Arney, J. Scott Yaruss, Hope Gerlach-Houck, Caryn Herring, Jia BinNAACL 2024 · DOI

TL;DR

This paper quantifies the significant accuracy bias of leading Automatic Speech Recognition (ASR) systems against disfluent speech, particularly that of individuals who stutter. By evaluating six popular ASRs on both real and synthetic disfluent speech datasets, the study reveals consistent and statistically significant performance degradation, highlighting a critical gap in current ASR technology for inclusive communication.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are becoming ubiquitous, powering voice assistants, dictation software, and numerous other applications. However, these systems are largely trained on "fluent" speech, leading to significant challenges in accurately transcribing speech patterns that deviate from this norm. This is particularly problematic for the estimated 80 million people worldwide who stutter. Stuttering is characterized by involuntary speech disruptions like repetitions, prolongations, and blocks, which current ASRs often struggle to interpret. This can lead to misinterpretations, usability issues, and even discrimination, especially in contexts like employment where voice-based technologies are increasingly prevalent. The paper aims to quantify this bias and understand its extent across different ASR models and disfluency types.

Methods

The researchers employed a two-pronged experimental approach to evaluate ASR performance on disfluent speech:

  • Experiment 1: Real Disfluent Speech: They used the FluencyBank dataset, a collection of real speech samples from individuals who stutter, including transcripts. To establish a baseline, they also created a "disfluency-removed" version of this dataset using text-to-speech (TTS) synthesis.
  • Experiment 2: Synthetic Disfluent Speech: They generated a synthetic disfluent speech dataset by introducing controlled disfluency events (repetitions, prolongations, blocks, interjections) into clean speech samples from the widely-used LibriSpeech dataset. This synthetic dataset was created using both textual disfluency insertion (modifying transcripts and using TTS) and audio disfluency insertion (directly manipulating audio waveforms).

They evaluated six popular ASR models (four commercial: Google Cloud, IBM Watson, Microsoft Azure, RevAI; and two open-source: wav2vec 2.0, Whisper) using three key metrics:

  • Word Error Rate (WER): A standard measure of transcription accuracy at the word level.
  • Character Error Rate (CER): A more granular measure of transcription accuracy at the character level.
  • BERTScore: A metric to evaluate the semantic similarity between the ASR-produced transcript and the reference transcript, crucial for understanding if the meaning is preserved despite errors.

Spearman correlation coefficients were used to analyze the relationship between ASR performance metrics on disfluent versus non-disfluent speech.

Datasets

  • FluencyBank: A real-world dataset of stuttered speech samples with accompanying transcripts, sourced from individuals who stutter.
  • FB-Y: The original FluencyBank dataset with disfluencies.
  • FB-N: A modified version of FluencyBank with apparent instances of stuttering removed, created using TTS.
  • LibriSpeech: A large, publicly available dataset of read English speech, commonly used for ASR training and evaluation.
  • LS-N: The "clean" subset of the LibriSpeech test set, representing non-disfluent speech.
  • LS-Y: A synthetic version of the LibriSpeech test set with controlled disfluency events introduced.

Key Insights and Ideas

  • Ubiquitous Bias: The study found a consistent and statistically significant accuracy bias across all tested ASR models against disfluent speech, regardless of whether the speech was real or synthetically generated. This suggests that current ASR architectures and training methodologies are inherently less effective with disfluent speech.
  • Disfluency Type Matters: Different types of disfluencies have varying impacts on ASR performance. Prolongations and interjections, for instance, showed a stronger correlation with increased word error rates compared to interword blocks. This highlights the need for nuanced approaches to handling various disfluency manifestations.
  • Semantic Drift: Beyond simple word errors, the study also found that disfluencies can lead to a degradation in semantic accuracy, meaning the transcribed text may not convey the intended meaning, even if individual words are somewhat correct.
  • Synthetic Data's Utility: The creation of a synthetic disfluent speech dataset proved valuable. It not only confirmed the biases observed with real data but also allowed for a controlled, granular analysis of how specific disfluency types affect ASR performance.

Key Results

  • All six evaluated ASR models showed lower accuracy (higher WER and CER) and poorer semantic similarity (lower BERTScore) on datasets containing disfluent speech compared to their non-disfluent counterparts.
  • For example, the Whisper model, considered state-of-the-art, experienced a significant drop in performance (WER from 3.1% to 19.8%) when disfluencies were introduced in the synthetic dataset.
  • The study found that prolongations and interjections had a particularly strong negative impact on WER.
  • Spearman correlation coefficients indicated statistically significant relationships between the presence of disfluencies and degraded ASR performance across most disfluency types and models.

Limitations and Future Work

Limitations: * The study focused on English speech and English-speaking individuals who stutter, not encompassing multilingual ASR models or speakers of other languages. * The synthetic dataset, while diverse, does not capture all possible types of disfluencies, and some disfluencies might not be considered stuttering in all contexts. * The study primarily assessed bias through accuracy metrics (WER, CER, semantic accuracy) and did not explore other potential bias facets like age or gender.

Future Work: * Diversify Training Data: Incorporating speech from people who stutter into ASR training data is crucial, though challenging due to data availability and the need to capture variability. * Human Feedback: Gathering ground-truth labels and preferences directly from individuals who stutter could lead to more authentic and respectful ASR development. * Longer-Form Audio: Investigating the impact of longer audio segments on ASR biases, as current models may struggle with segmentation and transcription of extended disfluent speech. * Benchmarking and Feedback Loops: Establishing rigorous testing protocols for disfluent speech before ASR deployment and creating feedback mechanisms for continuous improvement.

Summary updated

BibTeX
@inproceedings{mujtaba2024lost,
  author = {Dena F. Mujtaba and Nihar R. Mahapatra and Megan Arney and J. Scott Yaruss and Hope Gerlach-Houck and Caryn Herring and Jia Bin},
  title = {{Lost in Transcription: Identifying and Quantifying the Accuracy Biases of Automatic Speech Recognition Systems Against Disfluent Speech}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.269},
}
Figure