How Might We Create Better Benchmarks for Speech Recognition?

Alëna Aksënova, Daan van Esch, James Flynn, Pavel GolikACL 2021

TL;DR

Current Automatic Speech Recognition (ASR) benchmarks, while showing impressive improvements, fail to capture real-world performance across diverse use cases and linguistic variations. This paper proposes a more comprehensive approach to ASR benchmarking by introducing a taxonomy of use cases, advocating for a wider range of metrics beyond Word Error Rate (WER), and emphasizing the importance of demographic metadata for fairer and more representative evaluations.

What Problem Does This Paper Address?

The field of Automatic Speech Recognition (ASR) has seen remarkable progress, with state-of-the-art systems achieving very low Word Error Rates (WER) on standard benchmarks like Librispeech. However, this success doesn't always translate to real-world applications. Systems that perform exceptionally well on these benchmarks often struggle to generalize to different use cases, domains, or linguistic variations. This gap arises because current benchmarks are too narrow, focusing primarily on read speech and a limited set of metrics (mainly WER). This can lead to a false sense of progress and hinder the development of ASR systems that are truly robust and equitable across diverse user populations and scenarios. The paper argues that this over-reliance on a single metric and a limited scope of evaluation is akin to Goodhart's Law, where a measure that becomes a target ceases to be a good measure.

Methods

The authors propose a multi-faceted approach to improving ASR benchmarks:

  • Taxonomy of ASR Use Cases: They categorize ASR applications into "horizontals" (e.g., dictation, voice search, conversations, audiobooks) and "verticals" (technical challenges like domain-specific terminology, speech rate, acoustic environment, encoding formats). This provides a structured way to think about the diverse environments and tasks ASR systems encounter.
  • Beyond WER Metrics: The paper advocates for a richer set of evaluation metrics beyond WER. This includes:
  • Latency: Crucial for real-time applications, with nuances in how it's measured (e.g., end-to-end vs. pure ASR latency).
  • Real-Time Factor (RTF): Measures processing speed relative to audio duration, important for streaming ASR.
  • Metadata about Words: Beyond transcription, metrics like word-level confidence scores, timestamps, and phonemic transcriptions are vital for downstream applications.
  • Hallucination Rate: Explicitly measuring instances where the ASR system generates text for silence or out-of-domain audio.
  • Debuggability and Fixability: A qualitative but important aspect, assessing how easily systems can be updated or corrected.
  • Demographically Informed Quality: A key contribution is the emphasis on incorporating demographic metadata (e.g., accent, gender, age, dialect, sociolect, L2 background, speech impairments) into benchmark evaluations. This allows for "population-weighted slicing," where performance is analyzed across different demographic groups, revealing potential biases or performance gaps that aggregate metrics might mask. They propose a visualization framework for this.

Datasets

The paper references several existing datasets commonly used in ASR research, including:

  • Librispeech: A popular benchmark for read speech.
  • TIMIT, Switchboard, WSJ, CALLHOME, Fisher: Other established datasets covering various speech types.
  • Mozilla Common Voice: Noted for its multilingual data and dialect/accent tags.
  • CORAAL: Specifically for AAVE (African American Vernacular English).
  • ESLORA: For Galician Spanish.
  • Corpus Gesproken Nederlands: For Dutch.
  • Aurora corpora: Used for evaluating performance in noisy conditions.

The paper does not introduce a new benchmark dataset but rather proposes a framework for how existing and future datasets should be structured and evaluated.

Key Insights and Ideas

  • The "Low Single Digits" Illusion: Achieving very low WER on current benchmarks doesn't guarantee real-world robustness. Small improvements are becoming harder to achieve and may be concentrated on specific error types (e.g., filler words).
  • Generalization is Key: ASR systems trained on one domain (like Librispeech) can fail spectacularly on others, even producing nonsensical output, highlighting the need for broader evaluation.
  • WER is Not Enough: While a foundational metric, WER alone is insufficient. Latency, processing speed, and the accuracy of associated metadata (like timestamps) are critical for practical deployment.
  • Demographic Bias is a Major Concern: ASR systems can exhibit significant performance disparities across different demographic groups (accents, dialects, gender, etc.). Benchmarks must account for this to ensure fairness and equity.
  • Population-Weighted Slicing: A powerful concept for understanding ASR performance by analyzing metrics across demographic groups, re-weighting them according to real-world population distributions to identify and address performance gaps.

Key Results

The paper doesn't present quantitative results from a new benchmark but rather outlines a framework for future evaluation. The "key results" are conceptual and methodological:

  • Proposed Taxonomy: A structured way to categorize ASR use cases and technical challenges.
  • Expanded Metric Set: A comprehensive list of metrics beyond WER, including latency, RTF, word metadata, and hallucination rates.
  • Demographic Analysis Framework: A methodology for evaluating ASR systems across various demographic groups, illustrated with hypothetical examples (Figure 1). This framework aims to reveal performance disparities that aggregate metrics would miss.

Limitations and Future Work

  • Data Sparsity: Acknowledged challenge in obtaining sufficient data for all demographic slices, especially when considering intersections of multiple demographic factors.
  • Complexity of Organization and Presentation: Implementing a benchmark with the proposed breadth of metrics and data slices will require significant effort in organizing, presenting, and understanding the material. The authors suggest "system report cards" as a potential way to capture this information.
  • Interdisciplinary Collaboration: Developing truly representative benchmarks requires collaboration between linguists and ASR developers to understand sociolinguistic variation.
  • Future Work: The authors explicitly call for future work in developing benchmarks that incorporate the proposed taxonomy, metrics, and demographic considerations. They believe this is a long-term journey crucial for the advancement of the ASR field.

Summary updated

BibTeX
@inproceedings{aksenova2021how,
  author = {Alëna Aksënova and Daan van Esch and James Flynn and Pavel Golik},
  title = {{How Might We Create Better Benchmarks for Speech Recognition?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure