Low-resource speech recognition and dialect identification of Irish in a multi-task framework

Liam Lonergan, Mengjie Qian, Neasa Ní Chiaráin, Christer Gobl, Ailbhe Ní Chasaide

TL;DR

This paper introduces a novel multi-task learning framework using hybrid CTC/Attention models with Intermediate CTC (InterCTC) to simultaneously improve Automatic Speech Recognition (ASR) and Dialect Identification (DID) for low-resource Irish. The proposed approach achieves a significant 10.8% relative improvement in DID accuracy and narrows the performance gap for ASR compared to existing state-of-the-art models.

What Problem Does This Paper Address?

Developing effective speech technology for low-resource languages like Irish presents a dual challenge: limited data hinders accurate ASR, and significant dialectal variation further complicates recognition. Traditional approaches often treat these as separate problems or rely on extensive data balancing, which can be insufficient. This paper tackles the problem of improving both ASR and DID for Irish by leveraging their inherent relationship within a unified multi-task learning framework, aiming to create more robust and adaptable models.

Methods

The core of this work is a hybrid CTC/Attention encoder-decoder architecture. The authors explore the use of Intermediate CTC (InterCTC), a technique that applies CTC loss to intermediate layers of the encoder. This is used as an auxiliary task to help the model learn dialect-specific features. They systematically investigate the optimal placement of InterCTC objectives within the encoder.

Two encoder architectures are evaluated: the Conformer and the more recent E-branchformer. Both are pre-trained on the multilingual XLS-R 300M model. For ASR, a standard transformer decoder is used. For DID, dialect tags are prepended to the text output, and a separate DID InterCTC objective is introduced. Finally, they explore multi-task language model (LM) shallow fusion, where a transformer LM is fine-tuned with dialect information and combined with the acoustic model.

Datasets

The experiments utilize several Irish speech corpora:

  • MíleGlór: A crowd-sourced corpus for ASR development.
  • Synthesis: Recordings used for generating synthetic voices.
  • Audiobooks: Recordings from various sources, including home recordings and published works.
  • Corpas na Cainte Beo: A spontaneous speech corpus from the New English-Irish Dictionary project, tagged with dialect information.

For LM training and fine-tuning, the following text corpora were used:

  • Paracrawl & ConLL17: Large web-scraped corpora used for pre-training the LM.
  • Historical Irish Corpus: Texts from 1900-1950, rich in dialectal markers.
  • Spontaneous Speech Training Transcripts: Transcripts from the "Corpas na Cainte Beo" corpus.

Key Insights and Ideas

  • InterCTC for Dialect: Intermediate CTC proves to be a valuable tool for explicitly guiding the model to learn dialectal features, especially when applied strategically to specific encoder layers.
  • Multi-task Synergy: Jointly training for ASR and DID, particularly with an optimized InterCTC setup, can lead to significant improvements in DID accuracy without drastically sacrificing ASR performance.
  • E-branchformer Potential: The E-branchformer architecture shows promise, with the larger version achieving competitive results, especially when combined with multi-task learning.
  • LM Shallow Fusion Benefits: Fine-tuning a language model with dialect information and fusing it with the acoustic model offers a modest but beneficial improvement in DID accuracy and a more substantial reduction in WER.

Key Results

  • The best performing model, an E-branchformer Large encoder with an optimized InterCTC configuration and LM shallow fusion, achieved a DID accuracy of 81.5%. This represents a 10.8% relative improvement over the baseline ECAPA-TDNN model (72.5% accuracy).
  • The ASR performance (WER) of the proposed multi-task models approached that of the traditional TDNN-HMM baseline, with the best model achieving 12.3% WER (compared to the TDNN-HMM baseline of 13.2% WER).
  • While jointly training for ASR and DID without InterCTC led to ASR performance degradation, the optimal InterCTC setup significantly mitigated this issue.

Limitations and Future Work

  • InterCTC's Impact on DID: The paper notes that the overall contribution of InterCTC to DID accuracy is slight, with improvements primarily seen in specific configurations. Further research could explore more advanced ways to leverage InterCTC for classification tasks.
  • ASR Performance Gap: While the gap is reduced, the end-to-end models still do not outperform the modular TDNN-HMM model for ASR. Future work could focus on further closing this gap.
  • LM Shallow Fusion Potential: The authors suggest that with more available dialect-tagged text corpora, the LM shallow fusion approach could yield even greater improvements.
  • Data Scarcity: The inherent low-resource nature of Irish remains a challenge. Continued efforts in data collection and augmentation are crucial.

Summary updated

Abstract

This paper explores the use of Hybrid CTC/Attention encoder-decoder models trained with Intermediate CTC (InterCTC) for Irish (Gaelic) low-resource speech recognition (ASR) and dialect identification (DID). Results are compared to the current best performing models trained for ASR (TDNN-HMM) and DID (ECAPA-TDNN). An optimal InterCTC setting is initially established using a Conformer encoder. This setting is then used to train a model with an E-branchformer encoder and the performance of both architectures are compared. A multi-task fine-tuning approach is adopted for language model (LM) shallow fusion. The experiments yielded an improvement in DID accuracy of 10.8% relative to a baseline ECAPA-TDNN, and WER performance approaching the TDNN-HMM model. This multi-task approach emerges as a promising strategy for Irish low-resource ASR and DID.
BibTeX
@inproceedings{lonergan2024low,
  author = {Liam Lonergan and Mengjie Qian and Neasa Ní Chiaráin and Christer Gobl and Ailbhe Ní Chasaide},
  title = {{Low-resource speech recognition and dialect identification of Irish in a multi-task framework}},
  booktitle = {Odyssey},
  year = {2024},
  url = {https://openreview.net/forum?id=neyWltOHgF},
}
Figure