Enenlhet as a case-study to investigate ASR model generalizability for language documentation

Éric Le Ferrand, Raina Heaton, Emily Prud’hommeauxNAACL 2024

TL;DR

This paper investigates how well state-of-the-art Automatic Speech Recognition (ASR) models, specifically Whisper and wav2vec, generalize to new speech data from an endangered language (Enenlhet) when fine-tuned on a limited corpus. The findings suggest that while ASR can provide a useful first pass for transcription, factors like vocabulary diversity and audio quality significantly impact accuracy, and models may not always generalize well to unseen data.

What Problem Does This Paper Address?

Documenting endangered languages is crucial for linguistic preservation, but the process is often hampered by a "transcription bottleneck." Manually transcribing vast amounts of audio data is time-consuming and labor-intensive, limiting the amount of data available for analysis or language instruction. Automatic Speech Recognition (ASR) offers a potential solution by generating initial transcriptions that can then be corrected by humans.

However, a major challenge in applying ASR to low-resource languages is the scarcity of transcribed data needed to train effective models. While fine-tuning large, multilingual ASR models on small datasets has shown promise, many existing demonstrations use data from a single project or under uniform conditions. This raises a critical question: how well do these fine-tuned models generalize to new data that might vary in recording quality, transcription conventions, or speaker population? This paper addresses this by using Enenlhet, an endangered Indigenous language spoken in Paraguay, as a case study to test ASR model generalizability.

Methods

The researchers employed two prominent ASR frameworks for fine-tuning: Whisper (Radford et al., 2022) and wav2vec (Baevski et al., 2020; Conneau et al., 2021). For Whisper, they used the "medium" model, and for wav2vec, they utilized the "xlsr-53" model. For wav2vec, they also incorporated a language model trained on the relevant training transcripts for decoding.

To simulate testing on new, unseen data, they implemented a "leave-one-out" cross-validation approach. For each speaker in their dataset, they trained an ASR model using all recordings except for that speaker's data, and then tested the model on the held-out speaker's recordings. This method directly addresses the question of generalization to new speakers and their data.

Datasets

The primary dataset used in this study is a corpus of speech recordings for the Enenlhet language (ISO 639-3 code: tmf). This language is spoken by fewer than 2,000 people in Paraguay and has minimal existing documentation.

  • Total recordings: Over 120 hours from more than 40 individuals.
  • Transcribed data: Approximately 10 hours of recordings from 16 speakers, with utterance-level timestamps.
  • Effective transcribed data for ASR training: Approximately 5 hours (after stripping silence, non-Enenlhet speech, and interviewer speech).

This dataset is notable for being relatively substantial for an endangered language and for having a large amount of untranscribed audio, offering potential for unsupervised training. The data was collected over multiple years under varying conditions and has been archived with the Archive of Indigenous Languages of Latin America (AILLA).

Key Insights and Ideas

  • Generalization is a real challenge: The study strongly suggests that ASR models fine-tuned on limited data may not always generalize well to new, unseen data, which could hinder their utility in language documentation.
  • Vocabulary diversity matters significantly: The Out-of-Vocabulary (OOV) rate, representing words in the test set not seen in the training set, was identified as the most impactful factor on Word Error Rate (WER). This highlights the importance of diverse lexical coverage in training data.
  • Audio quality is a double-edged sword: While loudness had minimal impact, "sharpness" (related to high-frequency content) negatively affected WER. This suggests that acoustic characteristics of recordings can significantly influence ASR performance.
  • CTC alignment as a proxy for quality: Posterior probabilities from CTC alignment showed promise in indicating transcription quality, with higher confidence correlating with lower WER. However, this measure's reliability is questioned when there are significant mismatches between transcription and audio (e.g., translation errors).
  • Whisper outperforms wav2vec: Across experiments, Whisper consistently achieved better performance (lower WER) than wav2vec.

Key Results

  • Whisper vs. wav2vec: Whisper generally performed better than wav2vec, exhibiting lower Word Error Rates (WER) across most speaker tests.
  • Leave-one-out vs. Random Split: The leave-one-out evaluation approach did not significantly underperform or overperform a random split approach for this specific dataset, suggesting that the inherent variability in the data itself is a primary driver of performance differences.
  • Impact of Factors on WER:
    • OOV Tokens & Types: Significantly and positively correlated with WER, meaning higher OOV rates lead to worse performance. This effect was more pronounced for wav2vec.
  • Training/Test Duration & Loudness: Found to have no significant impact on WER.
  • Sharpness: Negatively impacted WER (higher sharpness, worse performance).
  • CTC Alignment Confidence: Showed a strong negative correlation with WER, suggesting higher confidence implies better alignment and potentially lower WER.

Limitations and Future Work

  • Generalizability Concerns: The authors acknowledge that their results suggest ASR models may not always generalize well to new data in low-resource settings, which could limit their practical application in language documentation.
  • Audio Quality Measures: While sharpness showed promise, further analysis on larger datasets is needed to validate its efficacy in assessing audio quality for ASR.
  • CTC Alignment Interpretation: The reliability of CTC posterior probabilities as a measure of transcription quality needs further investigation, especially in cases of transcription errors or mismatches.
  • Future Work: The paper implicitly suggests the need for more robust methods to handle OOV words, improve audio quality assessment, and develop ASR systems that are more resilient to variations in data collection and speaker characteristics. Further research into data augmentation and more sophisticated fine-tuning strategies could also be beneficial.

Summary updated

BibTeX
@inproceedings{ferrand2024enenlhet,
  author = {Éric Le Ferrand and Raina Heaton and Emily Prud’hommeaux},
  title = {{Enenlhet as a case-study to investigate ASR model generalizability for language documentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure