Data Augmentation for Speech Recognition in Maltese: A Low-Resource Perspective

Carlos Daniel Hernandez Mena, Andrea DeMarco, Claudia Borg, Lonneke van der Plas, Albert GattarXiv 2021 · OpenReview

TL;DR

This paper explores data augmentation techniques to improve Automatic Speech Recognition (ASR) for Maltese, a low-resource language with limited training data. By combining unsupervised training, multilingual data, and synthesized speech, they achieved a significant 15% absolute Word Error Rate (WER) improvement, demonstrating the effectiveness of these methods for under-resourced languages.

What Problem Does This Paper Address?

Developing robust ASR systems typically requires vast amounts of transcribed speech data. However, many languages, like Maltese, lack such extensive resources, making it challenging to build high-performing ASR models. This paper tackles the problem of low-resource ASR by investigating various data augmentation strategies to effectively leverage limited available data and data from related, higher-resource languages. The goal is to find the most effective augmentation techniques when starting with a small, curated corpus (around 7 hours) of Maltese speech.

Methods

The researchers explored three primary data augmentation strategies:

  • Unsupervised Training: This involves using transcriptions generated by an ASR system itself, which are inherently noisy. They experimented with both noisy transcriptions of Maltese and transcriptions of other languages generated by a Maltese ASR model.
  • Multilingual Training: This strategy leverages data from typologically or historically related languages (Arabic, Italian, English) to improve Maltese ASR. They explored using both noisy transcriptions of these languages and, crucially, "gold" (human-verified) transcriptions from English.
  • Training with Synthesized Data: This method involves generating artificial speech data using a Text-to-Speech (TTS) system. Even though the synthesized data was acknowledged as imperfect and noisy, the researchers investigated its utility.

A key aspect of their methodology was a pretraining and fine-tuning pipeline. Data augmentation was applied during the pretraining phase, and the models were then fine-tuned on the limited, high-quality MASRI-Headset corpus. They also deliberately excluded language model (LM) rescoring to isolate the impact of data augmentation on the acoustic model itself.

Datasets

The primary dataset for evaluation and fine-tuning was the MASRI-HEADSET corpus, a curated collection of approximately 7 hours of read Maltese speech.

For data augmentation, they utilized:

  • MASRI-SYNTHETIC Corpus: Artificially generated Maltese speech using a diphone-based TTS system, creating approximately 99 hours of data.
  • Maltese audiobooks: Approximately 12 hours of untranscribed Maltese audio.
  • LibriSpeech: A large English ASR corpus (100 hours used).
  • Arabic Speech MGB-2 corpus: A portion of an Arabic broadcast news corpus (100 hours used).
  • M-AILABS Speech Dataset (Italian portion): Approximately 118 hours of Italian read speech.

Key Insights and Ideas

  • Noisy data can be surprisingly effective: Even imperfect, automatically generated transcriptions (unsupervised or synthesized) can significantly boost ASR performance, especially when target language data is scarce.
  • Gold standard data from related languages is powerful: Combining noisy Maltese data with high-quality, transcribed data from a related language like English yielded substantial improvements, highlighting the value of transfer learning.
  • Language mixtures offer benefits, but with caveats: Mixing data from multiple related languages improved performance, but the gains were not always additive and could be less effective than using a single, high-quality related language dataset.
  • Synthesized speech is a viable option: Despite its inherent noise and unnaturalness, synthesized Maltese speech provided significant gains, demonstrating that even low-quality synthetic data can be beneficial for pretraining.
  • Focus on augmentation over architecture complexity: The study suggests that for low-resource scenarios, the choice of data augmentation strategies can be more impactful than using overly complex model architectures.

Key Results

The paper reports a significant absolute 15% improvement in Word Error Rate (WER) when combining the most effective data augmentation techniques.

  • The baseline system (without augmentation) achieved a WER of 63.71%.
  • The best augmented system achieved a WER of 48.97%.
  • This improvement was achieved without the use of a language model, isolating the impact of data augmentation.
  • Pretraining with 100 hours of noisy data from non-target languages (English, Arabic) yielded improved WER results compared to using only Maltese data.
  • Mixing gold English transcriptions with noisy Maltese transcriptions resulted in a WER of around 55%, exceeding language mixture experiments.
  • Using synthesized Maltese speech alone brought the WER down to 52%, and combining it with gold English data further reduced it to 48.97%.

Limitations and Future Work

  • Limited Maltese data: The study acknowledges the inherent limitation of working with a very small amount of high-quality Maltese data (around 7 hours).
  • Noisy synthesized data: The TTS system used produced imperfect and noisy speech, which, while beneficial, still represents a limitation.
  • Exclusion of Language Models: The paper deliberately excluded LM rescoring to focus on data augmentation's impact on acoustic models. However, the authors note that LM rescoring would likely further improve results.

Future work suggested includes:

  • Combining the best data augmentation strategies with more advanced neural architectures, particularly Transformer-based models like Wav2Vec 2.0.
  • Integrating language modeling into future experiments to achieve state-of-the-art performance for Maltese ASR.
  • Exploring the generalizability of these findings to other under-resourced languages.
  • Continuing efforts in crowd-sourced data collection for Maltese.

Summary updated

Abstract

Recent years have seen an increased interest in the computational speech processing of Maltese, but resources remain sparse. In this paper, we consider data augmentation techniques for improving speech recognition for low-resource languages, focusing on Maltese as a test case. We consider three different types of data augmentation: unsupervised training, multilingual training and the use of synthesized speech as training data. The goal is to determine which of these techniques, or combination of them, is the most effective to improve speech recognition for languages where the starting point is a small corpus of approximately 7 hours of transcribed speech. Our results show that combining the data augmentation techniques studied here lead us to an absolute WER improvement of 15% without the use of a language model.
BibTeX
@misc{mena2021data,
  author = {Carlos Daniel Hernandez Mena and Andrea DeMarco and Claudia Borg and Lonneke van der Plas and Albert Gatt},
  title = {{Data Augmentation for Speech Recognition in Maltese: A Low-Resource Perspective}},
  year = {2021},
  eprint = {2111.07793},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.07793},
}
Figure