Say What? A Dataset for Exploring the Error Patterns That Two ASR Engines Make

Meredith Moore, Michael Saxon, Hemanth Venkateswara, Visar Berisha, Sethuraman PanchanathanInterspeech 2019 · OpenReview

Abstract

We present a new metadataset which provides insight into where and how two ASR systems make errors on several different speech datasets. By making this data readily available to researchers, we hope to stimulate research in the area of WER estimation models, in order to gain a deeper understanding of how intelligibility is encoded in speech. Using this dataset, we attempt to estimate intelligibility using a state-of-the-art model for speech quality estimation and found that this model did not work to model speech intelligibility. This finding sheds light on the relationship between how speech quality is encoded in acoustic features and how intelligibility is encoded. It shows that we have a lot more to learn in how to effectively model intelligibility. It is our hope that the metadataset we present will stimulate research into creating systems that more effectively model intelligibility.

Introduction

In the field of Automatic Speech Recognition (ASR), a critical distinction exists between speech quality—how clear or pleasing a signal sounds—and intelligibility—how well the spoken words can be understood. While traditional metrics like the Articulation Index focus on signal degradation (e.g., background noise), they often fail to account for clear audio that remains unintelligible due to heavy accents or speech disorders.

Currently, measuring ASR performance requires a "ground truth" transcript to calculate the Word Error Rate (WER). This research aims to bridge the gap toward "non-intrusive" estimation, where intelligibility can be predicted from acoustic features alone without needing a transcript. To facilitate this, the authors have released a comprehensive metadataset mapping WER patterns across diverse speaker types.

Methodology and Data Sources

The researchers processed over 36,000 speakers across two distinct ASR engines: Google Speech Recognition (deep learning-based) and CMUSphinx (Hidden Markov Model-based). For every audio sample, the metadataset records the reference text, the ASR hypothesis, substitutions, insertions, deletions, final WER, and processing time.

To ensure a broad spectrum of intelligibility, the data was aggregated from five sources:

  • TIMIT: Healthy native speech (high intelligibility).
  • Mozilla Common Voice: Diverse, open-source data with realistic noise.
  • Speech Accent Archive: 2,140 individuals with varying native languages.
  • UASpeech & TORGO: Clinical datasets featuring dysarthric speech resulting from neurological injury.

Results: The Spectrum of ASR Performance

The metadataset captures a massive variance in difficulty across these populations. As shown in the performance analysis, average WER ranged from a low of 0.09 for native TIMIT speech to a high of 1.16 for disordered speech in the UA/TORGO datasets. Notably, WER can exceed 1.0 due to a high frequency of word insertions by the ASR engines.

Average WER across the five datasets included in the metadataset.

The "Quality vs. Intelligibility" Paradox

A key experiment involved "Intellinet," a model designed to test if state-of-the-art quality estimation architectures could predict intelligibility. Intellinet replicated the architecture of Quality-Net (a bidirectional LSTM used for PESQ scores) but was trained to predict WER.

The results revealed a fundamental disconnect. While the architecture is highly effective at predicting speech quality (achieving a Linear Correlation Coefficient of 0.90), it failed completely at predicting intelligibility (LCC of 0.023).

Comparison of model performance: (A) Quality-Net predicting PESQ scores vs. (B) Intellinet attempting to predict WER.

As illustrated in Figure 4B, the model suffered from the "Horizontal Line Problem," where it essentially predicted the global average WER for every sample regardless of the actual acoustics. This suggests that intelligibility is not merely encoded in frame-level acoustics but involves complex interactions between phonemes and linguistic context that current quality-centric models cannot capture.

Conclusion and Future Work

The failure of off-the-shelf quality models to predict WER proves that "clarity" and "understanding" are processed differently by ASR systems. The authors conclude that future intelligibility estimators must be multi-modal, integrating local acoustics, global signal characteristics, and linguistic data. By providing this pre-computed metadataset—which required six days of continuous computation to generate—the researchers hope to enable the community to develop more sophisticated models for real-time, non-intrusive intelligibility scoring.

Summary updated

Abstract

We present a new metadataset which provides insight into where and how two ASR systems make errors on several different speech datasets. By making this data readily available to researchers, we hope to stimulate research in the area of WER estimation models, in order to gain a deeper understanding of how intelligibility is encoded in speech. Using this dataset, we attempt to estimate intelligibility using a state-of-the-art model for speech quality estimation and found that this model did not work to model speech intelligibility. This finding sheds light on the relationship between how speech quality is encoded in acoustic features and how intelligibility is encoded. It shows that we have a lot more to learn in how to effectively model intelligibility. It is our hope that the metadataset we present will stimulate research into creating systems that more effectively model intelligibility.
BibTeX
@inproceedings{moore2019say,
  author = {Meredith Moore and Michael Saxon and Hemanth Venkateswara and Visar Berisha and Sethuraman Panchanathan},
  title = {{Say What? A Dataset for Exploring the Error Patterns That Two ASR Engines Make}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=aUafDLLZzIu},
}
Figure