Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect

Salima Mdhaffar, Haroun Elleuch, Fethi Bougares, Yannick EstèveACL 2024

TL;DR

This paper benchmarks various self-supervised learning (SSL) speech encoders for low-resource Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR) in Tunisian Arabic. They found that the w2v-BERT 2.0 model, trained on a massive multilingual dataset, offers the best performance for both tasks, highlighting the power of large-scale pre-training even for low-resource languages.

What Problem Does This Paper Address?

The field of speech processing has seen tremendous progress with self-supervised learning (SSL) models, which learn rich representations from vast amounts of unlabeled audio data. These models can then be fine-tuned for downstream tasks like Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR) with significantly less labeled data. However, most research has focused on high-resource languages like English. There's a critical gap in understanding how these powerful SSL models perform on low-resource languages and dialects, such as spoken Tunisian Arabic, where labeled data is scarce. This paper aims to bridge this gap by systematically evaluating a range of SSL speech encoders on this specific low-resource scenario, providing crucial insights for developing effective speech technologies for under-represented languages.

Methods

The core of this research is a comprehensive benchmarking study. The authors evaluated numerous SSL speech encoders, including both monolingual and multilingual models. They also explored the impact of a teacher-student approach for refining semantic representations. The experiments involved fine-tuning these pre-trained encoders on the TARIC-SLU dataset for two primary tasks:

  • Automatic Speech Recognition (ASR): Transcribing spoken audio into text.
  • Spoken Language Understanding (SLU): Extracting semantic information from speech, specifically focusing on Speech Act classification (identifying the speaker's intent) and Slot Filling (extracting specific pieces of information like locations or times).

They employed standard evaluation metrics for these tasks: Word Error Rate (WER) for ASR and SLU transcripts, and Speech Act Error Rate (SAER), Concept Error Rate (COER), and Concept/Value Error Rate (CVER) for SLU.

Datasets

The primary dataset used in this study is TARIC-SLU. This dataset was collected from human-human conversations in Tunisian train stations and includes manual transcriptions and semantic annotations. It comprises over 2,000 dialogues from 109 speakers and is split into training, development, and testing sets. The dataset is annotated with 62 semantic concept tags and 3 speech acts.

Key Insights and Ideas

  • Multilingual Pre-training Dominates: The study strongly suggests that large-scale, multilingual pre-training is highly beneficial, even for low-resource languages like Tunisian Arabic. The w2v-BERT 2.0 model, trained on a vast amount of data across many languages, consistently outperformed others.
  • SSL for Low-Resource is Viable: Despite the scarcity of labeled data for Tunisian Arabic, SSL models demonstrate significant potential for both ASR and SLU tasks, offering a viable path for developing technologies for such languages.
  • Semantic Refinement Matters: While not always leading to the absolute best results, the teacher-student approach for semantic encoding (as seen in SAMU-XLSR) shows promise in improving semantic understanding, especially for more complex utterances.
  • Data2vec 2.0's Semantic Strength: While not the best for ASR transcription, data2vec 2.0 (trained on English) showed competitive performance in SLU's COER evaluation, indicating its ability to capture semantic concepts.
  • Whisper's ASR Prowess, but SLU Limitations: The recently released Whisper models performed well in ASR, often outperforming many SSL models. However, they fell short in providing accurate semantic extraction compared to the best SSL models.

Key Results

The paper reports several key quantitative findings:

  • Best ASR Performance: The w2v-BERT 2.0 model achieved the best ASR performance with a WER of 32.29% on the development set and 26.7% on the test set.
  • Best SLU Performance: w2v-BERT 2.0 also led in SLU performance, showing the best results across most metrics, including COER.
  • SONAR's Advantage: Among the SONAR models, SONAR-ARB demonstrated superior performance for both ASR and SLU tasks.
  • Whisper vs. SSL: Whisper-medium achieved competitive ASR results (33.96% WER on dev, 29.56% on test), outperforming most SSL models except for w2v-BERT 2.0, SAMU-XLSR, and WavLM. However, its semantic extraction capabilities were weaker.

Limitations and Future Work

  • Unavailability of w2v-BERT: The specific w2v-BERT model used to initialize the SONAR models was not publicly available, preventing a full assessment of its contribution.
  • SAMU-XLSR Availability: SAMU-XLSR was also not publicly available at the time of publication, though the authors received it from the researchers.
  • Future Work: The authors plan to train a SAMU-w2vBERT 2.0 model to leverage the combined strengths of SAMU's semantic enrichment and w2v-BERT 2.0's powerful representations. They also aim to make their code and models publicly available for reproducibility and to encourage further research in Arabic dialect SLU and ASR.

Summary updated

BibTeX
@inproceedings{mdhaffar2024performance,
  author = {Salima Mdhaffar and Haroun Elleuch and Fethi Bougares and Yannick Estève},
  title = {{Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure