Enhance Contextual Learning in ASR for Endangered Low-resource Languages

Zhaolin Li, Jan NiehuesNAACL 2025

TL;DR

This paper investigates how to improve Automatic Speech Recognition (ASR) for endangered low-resource languages by enhancing contextual learning through language model (LM) integration. They find that fine-grained tokenization is crucial, but surprisingly, transformer-based LMs don't consistently outperform traditional N-gram LMs in this setting, highlighting the unique challenges of low-resource languages.

What Problem Does This Paper Address?

Documenting endangered languages is vital, and ASR offers a powerful tool for this. However, ASR systems struggle with low-resource languages due to a lack of sufficient labeled data. While acoustic modeling has advanced significantly, effectively integrating linguistic knowledge through language models (LMs) to improve contextual understanding in ASR for these languages remains an underexplored area. Existing methods often rely on word-level LM integration, which falters with the high prevalence of unknown words in low-resource settings. Furthermore, the effectiveness of advanced transformer-based LMs, which excel in high-resource languages, is not well understood for these low-resource scenarios.

Methods

The core of this research lies in exploring different strategies for integrating language models into ASR systems for low-resource languages. The authors leverage a Wav2Vec2 acoustic model fine-tuned with CTC loss, a common approach for low-resource ASR.

Key methodological contributions include:

  • Fine-grained Tokenization: Instead of traditional word-level tokenization, they experiment with subword and character-level tokenization. This is achieved by encoding transcripts with special markers to denote word boundaries, allowing the ASR and LM to better handle unknown words and improve data utilization.
  • LM Integration: They integrate LMs during the decoding phase in an auto-regressive manner. The total score for a candidate text is a combination of the acoustic model's output and the LM's score, controlled by parameters $\alpha$ and $\beta$.
  • Comparison of LM Architectures: The study directly compares the performance of traditional N-gram LMs with more modern transformer-based LMs (specifically, GPT-2) for LM integration.
  • Language-Specific Optimization: They emphasize the importance of tuning LM integration parameters (like $\alpha$ and $\beta$) and other hyperparameters on a per-language basis, rather than transferring settings from one language to another.

Datasets

The study uses five diverse low-resource languages for experiments:

  • Khinalug: A Northeast Caucasian language.
  • Kichwa: A Quechuan language.
  • Mboshi: A Bantu language.
  • Japhug: A Sino-Tibetan language.
  • Bemba: A Bantu language (included to study the impact of additional supervised data).

The paper provides descriptive statistics for these datasets, including audio source, training hours, and the percentage of unknown words in development and test sets, highlighting the challenges posed by data scarcity and linguistic diversity.

Key Insights and Ideas

  • Tokenization is Key: Fine-grained tokenization (subword and character) significantly improves ASR performance by better handling unknown words and making more efficient use of available data. This is a crucial step for low-resource languages.
  • Transformer LMs Don't Always Win: Surprisingly, transformer-based LMs do not consistently outperform N-gram LMs in this low-resource ASR context, even though they excel in language modeling tasks. This suggests that the benefits of advanced LM architectures might be hindered by the limited data and specific linguistic characteristics of these languages.
  • Language-Specific Tuning is Non-Negotiable: Parameter transfer between languages leads to substantial performance degradation (up to 43.2%). This strongly indicates that optimal LM integration and ASR system configuration require careful, language-specific optimization.
  • Character Density Matters: For languages like Mboshi, high character density in speech can negatively impact character-level tokenization, leading to information loss and poorer performance.

Key Results

  • Tokenization Gains: Fine-grained tokenization (subword and character) led to relative Word Error Rate (WER) reductions of up to 9.8% compared to coarse word-level tokenization for Kichwa, Mboshi, Japhug, and Bemba.
  • LM Performance: Transformer-based LMs only outperformed N-gram LMs in one out of five languages (Japhug).
  • Parameter Transfer Penalty: Transferring optimized LM integration parameters from Kichwa to Mboshi resulted in a 32.0% performance degradation, and to Japhug, a 43.2% degradation.
  • Character Tokenization: Character-level tokenization generally yielded the best performance among the fine-grained approaches.

Limitations and Future Work

  • Mboshi Anomaly: The study notes that the Mboshi ASR model with character-level tokenization performed worse than expected. They attribute this to high character density and potentially morphological complexity, but acknowledge a lack of linguistic expertise to fully evaluate the latter.
  • LM Integration Nuances: The paper highlights that while transformer LMs might have lower perplexity (better language modeling), their ranking of candidates doesn't always align with the ASR gold transcript, suggesting current integration methods might not fully leverage their potential.
  • Future Directions: The authors suggest further investigation into the differences between transformer and N-gram LM integration for low-resource languages. They also point to the need for more research into language-specific optimization strategies and how to best adapt advanced LM architectures to the unique challenges of endangered languages.

Summary updated

BibTeX
@inproceedings{li2025enhance,
  author = {Zhaolin Li and Jan Niehues},
  title = {{Enhance Contextual Learning in ASR for Endangered Low-resource Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure