Towards spoken dialect identification of Irish

Liam Lonergan, Mengjie Qian, Neasa Ní Chiaráin, Christer Gobl, Ailbhe Ní ChasaidearXiv 2023 · OpenReview

TL;DR

This paper tackles the challenge of accurately identifying spoken dialects of Irish, a low-resource language with significant dialectal variation that hinders automatic speech recognition (ASR). By developing and fusing acoustic and text-based dialect identification models, they achieve a significant improvement in accuracy, paving the way for more equitable ASR systems for Irish speakers.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often perform poorly on non-standard speech varieties, including dialects. This is particularly problematic for languages like Irish, which lacks a single spoken standard and instead has three major native dialects: Ulster (Ul), Connacht (Co), and Munster (Mu), along with numerous sub-dialects. Previous research has shown that even with dialect-balanced training data, ASR systems can exhibit performance bias, with the Ulster dialect often being recognized less accurately. This disparity highlights the need for dedicated dialect identification (DID) systems that can either route speech to dialect-specific ASR models or provide valuable metadata to improve overall ASR performance. The authors aim to build such a DID system for Irish, addressing the low-resource nature of the language and the limited availability of dialect-tagged data.

Methods

The researchers explored two primary approaches for acoustic-based dialect identification:

  • XLS-R: A large, multilingual self-supervised speech representation model. They fine-tuned this model by adding a classification head.
  • ECAPA-TDNN: A state-of-the-art acoustic model architecture known for its effectiveness in speaker verification and language identification. They experimented with both training this model from scratch and fine-tuning a version pre-trained on the VoxLingua107 dataset (a large language identification dataset).

For text-based dialect identification, they utilized gaBERT, a BERT language model pre-trained on Irish text.

To leverage the strengths of both acoustic and text modalities, they implemented a model fusion technique. This involved combining the output probabilities of the acoustic and text models using a weighted average, where the optimal weight (λ) was determined through a grid search on a validation set.

Datasets

The study utilized a combination of acoustic and text datasets for training and evaluation:

  • Acoustic Data:
    • Training: Approximately 136 hours of speech data, comprising ~16 hours of carefully read speech and ~30 hours of spontaneous speech per dialect. This data was sourced from in-house recordings and a broadcast corpus provided by Foras na Gaeilge's New English-Irish Dictionary project.
  • Validation and Test: 1.5 hours and 2.5 hours, respectively, of read speech from native Irish speakers, with careful attention paid to balancing gender and dialect representation.
  • Text Data:
    • Training: Dialectal texts written between 1900-1950 from the Historical Irish Corpus (pre-standardization) and transcriptions from the spontaneous speech corpus.
  • Evaluation: Transcriptions from the acoustic test set.

Key Insights and Ideas

  • Dialectal Variation is a Significant Challenge: The experiments confirm that distinguishing between Irish dialects is a non-trivial task, especially for the Connacht and Munster dialects, which exhibit acoustic similarities.
  • Pre-trained Models Offer a Strong Starting Point: Fine-tuning a pre-trained ECAPA-TDNN model (originally trained for language identification) significantly outperformed training from scratch, highlighting the value of leveraging existing knowledge.
  • Textual Cues Complement Acoustic Information: While acoustic models performed reasonably well, fusing them with a text-based model provided a notable boost in accuracy, particularly for the more challenging dialect pairs.
  • Ulster Dialect is More Distinct: The Ulster dialect was consistently identified with much higher accuracy, suggesting it possesses more unique acoustic characteristics compared to Connacht and Munster.
  • The Need for Nuance: The struggle to disambiguate Connacht and Munster dialects suggests that future work might need to incorporate more linguistically informed features or explore more complex modeling approaches.

Key Results

  • Acoustic Models:
    • XLS-R achieved 55% accuracy.
  • ECAPA-TDNN trained from scratch achieved 64% accuracy.
  • Pre-trained ECAPA-TDNN (fine-tuned) achieved the highest acoustic accuracy at 73%.
  • Text-Based DID:
    • The gaBERT model achieved 87% accuracy on its original training domain but dropped to 54% on transcriptions from the acoustic test set, indicating a domain mismatch.
  • Fused Model:
    • Fusing the pre-trained ECAPA-TDNN with the text-based model (with an optimal weight of λ=0.2) improved the overall accuracy to 76%.
  • The Ulster dialect was identified with a remarkable 94% accuracy.
  • The fused model showed a 6% increase in accuracy for both Connacht and Munster dialects, though performance for Ulster dropped slightly.

Limitations and Future Work

  • Domain Mismatch in Text Data: The text-based classifier performed significantly worse on transcriptions from the acoustic test set compared to its original training data, highlighting a need for better alignment between text and speech domains.
  • Difficulty Disambiguating Southern Dialects: The model struggled to differentiate between Connacht and Munster dialects, suggesting that more nuanced approaches or linguistically salient features might be necessary.
  • Future Directions:
    • Explore the use of self-supervised models as feature extractors within the ECAPA-TDNN framework.
  • Investigate the incorporation of DID directly into the ASR pipeline to improve dialect performance equity.
  • Further research into linguistically salient dialect markers to improve disambiguation of closely related dialects.

Summary updated

Abstract

The Irish language is rich in its diversity of dialects and accents. This compounds the difficulty of creating a speech recognition system for the low-resource language, as such a system must contend with a high degree of variability with limited corpora. A recent study investigating dialect bias in Irish ASR found that balanced training corpora gave rise to unequal dialect performance, with performance for the Ulster dialect being consistently worse than for the Connacht or Munster dialects. Motivated by this, the present experiments investigate spoken dialect identification of Irish, with a view to incorporating such a system into the speech recognition pipeline. Two acoustic classification models are tested, XLS-R and ECAPA-TDNN, in conjunction with a text-based classifier using a pretrained Irish-language BERT model. The ECAPA-TDNN, particularly a model pretrained for language identification on the VoxLingua107 dataset, performed best overall, with an accuracy of 73%. This was further improved to 76% by fusing the model's outputs with the text-based model. The Ulster dialect was most accurately identified, with an accuracy of 94%, however the model struggled to disambiguate between the Connacht and Munster dialects, suggesting a more nuanced approach may be necessary to robustly distinguish between the dialects of Irish.
BibTeX
@misc{lonergan2023towards,
  author = {Liam Lonergan and Mengjie Qian and Neasa Ní Chiaráin and Christer Gobl and Ailbhe Ní Chasaide},
  title = {{Towards spoken dialect identification of Irish}},
  year = {2023},
  eprint = {2307.07436},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.07436},
}
Figure