Towards Variability Resistant Dialectal Speech Evaluation

Ahmed Ali, Salam Khalifa, Nizar Habash

Introduction

Evaluating Automatic Speech Recognition (ASR) systems for dialects is notoriously difficult due to the lack of standardized orthography. In languages like Egyptian Arabic, spelling is often phonetic or spontaneous, leading to inconsistent Word Error Rate (WER) scores. This research demonstrates that by using morphological abstraction and standardized orthography, we can create evaluation metrics that align much more closely with human judgment, even when using only a single reference transcript.

The Challenge of Dialectal Orthography

In standardized languages, spelling is fixed. However, for dialects, two human transcribers might hear the same audio and produce different—but equally valid—text. This creates a "gold standard" crisis where standard WER becomes "noisy," penalizing systems for valid orthographic variations. This paper seeks a "variability resistant" version of WER that ranks ASR systems the same way a human expert would.

Methodology

The researchers explored two main avenues to reduce spelling noise: Text Representation and Metric Variants.

Text Representations

Instead of comparing raw ASR output to raw human transcripts, they transformed text into more abstract forms:

  • CODA (Conventional Orthography for Dialectal Arabic): A proposed system to standardize dialectal spelling.
  • Morphological Tokenization (ATB and D3): These schemes strip clitics (like "and" or "the") from the base word.
  • Lemmatization (LEMMA): Reducing every word to its dictionary root.
  • Character-level (RAWchar): Evaluating based on individual characters rather than whole words.

Metric Variants

They compared standard single-reference WER against multi-reference variants like $WER_{avg}$ (average across references) and $WER_{mr}$ (matching any reference).

Key Findings

Inter-Annotator Disagreement

A primary insight is that even professional transcribers disagree on 9-21% of words in raw text. The study found that moving from raw text to standardized representations like CODA or Lemmatization significantly reduces this disagreement, providing a more stable baseline for evaluation.

Inter-annotator disagreement matrices for (a) Raw, (b) Normalized, (c) CODA, and (d) Lemmatized text.

Abstraction as a Cost-Saver

One of the most important findings is that using a more abstract representation (like D3 tokenization) with a single reference can be more effective than using multiple references with raw text. Since human transcription is expensive, this is a major efficiency gain for researchers.

D3: The "Sweet Spot"

While lemmatization is the most abstract, it can lose too much meaning. The D3 tokenization scheme emerged as the best balance, providing the highest correlation with human rankings. In contrast, character-level WER (RAWchar) produced lower error rates but had the worst correlation with human judgment of system quality.

Conclusion

By adopting morphological abstraction, specifically D3 tokenization, researchers can achieve ASR evaluation metrics that are both reliable and cost-effective. These metrics achieve up to 100% correlation with human rankings, providing a robust framework for developing dialectal speech technologies.

Summary updated

Abstract

We study the problem of evaluating automatic speech recognition (ASR) systems that target dialectal speech input. A major challenge in this case is that the orthography of dialects is typically not standardized. From an ASR evaluation perspective, this means that there is no clear gold standard for the expected output, and several possible outputs could be considered correct according to different human annotators, which makes standard word error rate (WER) inadequate as an evaluation metric. Specifically targeting the case of Arabic dialects, which are also morphologically rich and complex, we propose a number of alternative WER-based metrics that vary in terms of text representation, including different degrees of morphological abstraction and spelling normalization.We evaluate the efficacy of these metrics by comparing their correlation with human judgments on a validation set of 1,000 utterances. Our results show that the use of morphological abstractions and spelling normalization produces systems with higher correlation with human judgment. We released the code and the datasets to the research community.
BibTeX
@inproceedings{ali2019towards,
  author = {Ahmed Ali and Salam Khalifa and Nizar Habash},
  title = {{Towards Variability Resistant Dialectal Speech Evaluation}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2692},
}
Figure