Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation

Martijn Bartelds, Nay San, Bradley McDonnell, Dan Jurafsky, Martijn Wieling

Introduction

While Automatic Speech Recognition (ASR) systems for high-resource languages like English and Mandarin have achieved near-human parity, the vast majority of the world’s 7,000+ languages remain underserved. Building robust ASR typically requires thousands of hours of human-transcribed audio, a resource that is prohibitively expensive and time-consuming to produce for minority languages, regional dialects, or endangered tongues. This paper addresses the "data scarcity" bottleneck by investigating how self-supervised models can be effectively fine-tuned using minimal local data combined with strategic data augmentation.

Methodology

The researchers utilized XLS-R, a massive multilingual model with 317 million parameters pre-trained on 436,000 hours of speech across 128 languages. The core objective was to determine if this general linguistic foundation could be adapted to specific low-resource contexts using three primary strategies:

  1. Self-Training (Teacher-Student): A "Teacher" model is first trained on a small set of manually transcribed data (as little as 24 minutes). This teacher then generates "pseudo-labels" for a larger pool of unlabeled audio. Finally, a "Student" model is trained on the combined set of manual and pseudo-labeled data.
  2. Text-to-Speech (TTS) Augmentation: For the language Gronings, the researchers employed an existing TTS system to generate synthetic audio from text-only sources, creating a supplemental training set from scratch.
  3. Continued Pre-Training (CPT): The team tested whether further pre-training the XLS-R model on the target language's raw, unlabeled audio before fine-tuning would provide a performance boost.

Experimental Setup

The study focused on four typologically diverse minority languages and variants:

  • Gronings & West-Frisian: West Germanic languages spoken in the Netherlands.
  • Besemah & Nasal: Malayo-Polynesian languages spoken in Sumatra, Indonesia.

Data was sourced from ongoing language documentation projects. To simulate extreme low-resource conditions, the researchers limited the manually transcribed data to a maximum of four hours, with sub-tests conducted on sets as small as 24 minutes.

Key Insights and Results

Augmentation vs. Computation

A primary finding of the research is that data augmentation (self-training and TTS) is significantly more effective and computationally efficient than Continued Pre-Training (CPT). While CPT is a common industry practice, it yielded only marginal gains compared to the double-digit improvements provided by augmentation strategies.

The Efficacy of Self-Training

Self-training consistently improved performance across all four languages. For models initialized with only 24 minutes of data, self-training led to a relative Word Error Rate (WER) reduction of up to 20.5%. The study observed that even when the initial "Teacher" model is mediocre due to data constraints, the "Student" model successfully extracts useful patterns from the imperfect pseudo-labels. However, the researchers noted diminishing returns; eventually, the errors in the teacher's transcriptions begin to "pollute" the student's learning process.

TTS as a High-Impact Tool

In the case of Gronings, TTS augmentation proved to be a "silver bullet." Utilizing synthetic audio resulted in a 25.5% relative reduction in WER. Remarkably, TTS-augmented models trained on just 24 minutes of real speech performed nearly as well as models trained on four times as much manual transcription.

Discussion and Limitations

While the results are promising for the future of minority language technology, several limitations remain:

  • Speaker Overfitting: The TTS system used was based on a single speaker, potentially causing the model to overfit to a specific voice rather than generalizing to the community.
  • Sociolinguistic Variables: The study did not account for internal dialectal variations or social groups, which may affect real-world deployment.
  • Unlabeled Data Limits: Self-training requires a pool of unlabeled audio, which may still be difficult to acquire for the most critically endangered languages.

Conclusion

This research demonstrates that functional speech tools can be developed for minority languages without the need for massive manual transcription efforts. By leveraging pre-trained multilingual models like XLS-R and focusing on augmentation techniques like self-training and TTS, researchers can achieve significant performance gains even with less than 30 minutes of transcribed audio. These findings provide a scalable roadmap for bringing ASR technology to the "long tail" of the world's languages.

Summary updated

BibTeX
@inproceedings{bartelds2023making,
  author = {Martijn Bartelds and Nay San and Bradley McDonnell and Dan Jurafsky and Martijn Wieling},
  title = {{Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.42},
}
Figure