What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations

Kavya Manohar, Leena G. PillaiEMNLP 2024 · DOI

TL;DR

Standard text normalization routines used to evaluate multilingual speech recognition (ASR) models are fundamentally flawed for Indic scripts. By stripping away essential vowel signs and diacritics, these "clean-up" tools artificially inflate performance metrics, creating a false sense of progress while destroying the linguistic integrity of the languages they aim to measure.

Introduction: The Normalization Trap

In Automatic Speech Recognition (ASR), researchers evaluate models by comparing transcriptions to a "ground truth" using Word Error Rate (WER). To ensure fairness, text normalization is applied to ignore trivial differences like punctuation or capitalization. However, most state-of-the-art models, including OpenAI’s Whisper and Meta’s MMS, utilize normalization routines designed primarily for English and Latin-based scripts.

These routines often treat Unicode "mark" class characters—which include accents and diacritics—as noise to be replaced with spaces. While this approach is effective for English or Finnish, it is catastrophic for Indic scripts (such as Hindi, Tamil, and Malayalam) and Thai. In these languages, these "marks" represent essential vowel signs (matras) and consonant joiners (viramas). Stripping them reduces complex words into nonsensical strings of isolated consonants, rendering evaluation metrics meaningless.

Methods and Linguistic Destruction

The authors conducted two types of evaluations to expose these pitfalls:

  1. Intrinsic Evaluation: They processed identical example sentences across six languages through Whisper’s standard normalization routine and measured the METEOR score (a text similarity metric) between the original and normalized versions.
  2. Extrinsic Evaluation: They measured the Word Error Rate (WER) of the Whisper-Small model on the Google FLEURS dataset, comparing raw transcripts against those processed with Whisper’s normalizer for both baseline and fine-tuned models.
Table showing the total loss of linguistic structure in Indic and Thai scripts after standard normalization.

The results of the intrinsic evaluation were stark. While English and Finnish maintained high similarity scores (0.97 and 0.95), Tamil, Malayalam, and Thai plummeted to 0.00. This indicates that the normalization process effectively destroyed the entire linguistic structure of those sentences.

The Word Error Rate Paradox

The study reveals a startling paradox: normalization makes ASR models look significantly better on paper while the resulting text is unreadable. This is particularly evident in Malayalam, where normalization breaks single complex words into multiple "words" consisting of single consonants. This over-segmentation increases the word count in the reference text, which mathematically dilutes the error rate.

Word Error Rate (WER) comparison for baseline and fine-tuned models, illustrating how normalization artificially lowers error rates.

As shown in the data, applying normalization to the baseline Malayalam model caused the WER to drop from 287.4% to 135.2%. This is not an improvement in model accuracy, but rather a mathematical artifact of the flawed normalization routine. Even with fine-tuned models, normalization provides an artificial 10-34% "improvement" in WER for Indic languages, compared to a negligible 3-5% for English.

Conclusion and Future Directions

This research highlights a systemic "Latin bias" in Unicode handling within ASR evaluation pipelines. Because Whisper’s normalization code is integrated into popular libraries like Hugging Face transformers, this flawed method has become a de facto standard, potentially misleading the research community regarding the true state of the art in South Asian languages.

The authors call for the development of linguistically informed normalization routines created in collaboration with native speakers. Until language-specific "cleaners" are developed that understand the fundamental role of diacritics in non-Latin scripts, multilingual ASR leaderboards for Indic languages should be viewed with significant skepticism.

Summary updated

BibTeX
@inproceedings{manohar2024what,
  author = {Kavya Manohar and Leena G. Pillai},
  title = {{What is lost in Normalization? Exploring Pitfalls in Multilingual ASR Model Evaluations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.607},
}

Loading related content...

Figure