How Important is a Language Model for Low-resource ASR?

Zoey Liu, Nitin Venkateswaran, Éric Le Ferrand, Emily Prud'hommeauxACL 2024 · DOI

Introduction

While modern end-to-end speech recognition often treats Language Models (LMs) as optional, this research demonstrates that they remain essential for low-resource languages. In high-resource Automatic Speech Recognition (ASR)—such as English or Mandarin—neural end-to-end architectures have become so powerful that traditional n-gram language models are often seen as an "afterthought" or skipped entirely. However, for the other 6,000+ languages on Earth, data remains the primary bottleneck.

Researchers often assume that if audio data is scarce, the system should be bolstered with as much text data as possible to build a larger LM. This study investigates whether this assumption holds true, specifically asking:

  1. Does using an n-gram LM actually help neural ASR performance in low-resource settings?
  2. Is there a direct relationship between the size of the LM and ASR accuracy when data is limited?

Methodology

The authors evaluated two distinct ASR frameworks to ensure findings were not architecture-specific:

  • Kaldi DNN: A hybrid system that inherently requires an LM for decoding.
  • Wav2Vec2 (XLSR-53): A state-of-the-art self-supervised model fine-tuned on small amounts of transcribed audio.

Three LM settings were compared:

  • NO_LM: Decoding without a language model (only possible for Wav2Vec2).
  • LM_BASE: A trigram LM trained only on the transcripts of the audio training data.
  • LM_LARGE: A trigram LM trained on transcripts plus additional external text (e.g., the Bible, news articles, or grammar books).

To validate results, the study employed mixed-effects regression modeling, controlling for variables like perplexity (PPL) and out-of-vocabulary (OOV) rates.

Dataset Overview

The study utilized a diverse set of nine "in the wild" small ASR corpora, spanning six language families. These ranged from relatively "larger" low-resource sets like Bemba (19h) to critically endangered sets like Kunwok (51m).

Summary of the nine low-resource language corpora used in the study, including audio duration and language model sizes.

The external text for LM_LARGE varied by language, often relying on religious texts or news, which frequently differ in domain from the everyday speech found in the audio recordings.

Results and Analysis

The experimental results highlight several critical trends in low-resource ASR performance across different architectures.

Word Error Rate (WER) results for Kaldi and Wav2Vec2 models across different language model configurations.

The LM is Not Optional

For every language tested, using any LM resulted in lower Word Error Rates (WER) compared to no LM at all. In Wav2Vec2 experiments, adding LM_BASE dropped the WER significantly. For example, in Swahili, the WER dropped from 31.74% to 25.92%. Even a small LM trained only on transcripts provides a vital "sanity check" for the acoustic model.

The "Large LM" Paradox

Counterintuitively, a bigger LM does not always guarantee better ASR. For the smallest datasets (like Hupa or Kunwok), increasing the LM size provided negligible or even negative returns. For Kréyol and Kunwok, LM_LARGE actually performed slightly worse or identical to LM_BASE in several tests, despite the inclusion of significantly more text data.

The Domain Mismatch Trap

The authors found that while LM_LARGE reduced the number of "unknown" words (OOV rate), it often significantly increased perplexity. Because the additional text (like the Bible) was linguistically distinct from the audio (everyday fieldwork), the LM became "confused," leading to higher error rates. The benefits of a larger LM only materialized when the audio corpus was already relatively substantial, such as in the case of Bemba (19h).

Discussion

The regression analysis confirmed a significant positive coefficient for LM size in Kaldi, meaning that—when controlling for other factors—simply throwing more out-of-domain text at the problem can actually increase the error rate. This suggests that for truly endangered languages, the acoustic model is the primary weak point, and audio data trumps text data in terms of impact on performance.

Conclusion

This research provides a practical recommendation for language communities and researchers: if resources are limited, prioritize collecting and transcribing more audio rather than hunting for external text to build a massive language model.

Future work should explore smart data selection methods to select or augment text that more closely matches the target audio's domain, ensuring that language models support rather than hinder the recognition of endangered languages.

Summary updated

BibTeX
@inproceedings{liu2024how,
  author = {Zoey Liu and Nitin Venkateswaran and Éric Le Ferrand and Emily Prud'hommeaux},
  title = {{How Important is a Language Model for Low-resource ASR?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-acl.13},
}
Figure