Abstract
Arabic is known to present unique challenges for Automatic Speech Recognition (ASR). On one hand, its rich linguistic diversity and wide range of dialects complicate the development of robust, inclusive models. On the other, current multilingual ASR models are compute-intensive and lack proper comprehensive evaluations. In light of these challenges, we distill knowledge from large teacher models into smaller student variants that are more efficient. We also introduce a novel human-annotated dataset covering five under-represented Arabic dialects for evaluation. We further evaluate both our models and existing SoTA multilingual models on both standard available benchmarks and our new dialectal data. Our best-distilled model’s overall performance (45.0% WER) surpasses that of a SoTA model twice its size (SeamlessM4T-large-v2, WER=47.0%) and its teacher model (Whisper-large-v2, WER=55.1%), and its average performance on our new dialectal data (56.9% WER) outperforms all other models. To gain more insight into the poor performance of these models on dialectal data, we conduct an error analysis and report the main types of errors the different models tend to make.
Introduction
Automatic Speech Recognition (ASR) for Arabic is a linguistic minefield due to diglossia, where Modern Standard Arabic (MSA) is used for formal writing while distinct regional dialects (Dialectal Arabic or DA) are used in daily life. Current State-of-the-Art (SoTA) multilingual models like Whisper and SeamlessM4T are "compute-heavy" giants that, while proficient in MSA, often struggle with the nuances of regional dialects. Furthermore, existing benchmarks are heavily biased toward MSA, leaving a gap in our understanding of how these models perform for speakers in regions like Algiers or Sana'a. This research addresses two main gaps: the lack of efficient models and the absence of comprehensive evaluation data for under-represented Arabic dialects.
Methodology
The authors employ Knowledge Distillation (KD), where a large, pre-trained "teacher" model (Whisper-large-v2) trains a smaller "student" model. The process involves:
- Pseudo-Labeling: Using the teacher to transcribe massive amounts of unlabeled Arabic audio.
- Filtering: Applying a Word Error Rate (WER) threshold to filter out low-quality transcriptions, ensuring the student doesn't learn the teacher's mistakes.
- Architecture: Initializing student models using "maximally spaced" layers from the teacher to retain structural knowledge.
- Loss Function: Combining standard transcription loss with Kullback-Leibler (KL) divergence to force the student to mimic the teacher's probability distribution.
Data and Evaluation
The study utilizes standard benchmarks (Common Voice, FLEURS, MGB) alongside a crucial new contribution: a human-annotated "In-House" dataset covering five under-represented dialects: Algerian (ALG), Jordanian (JOR), Palestinian (PAL), Emirati (UAE), and Yemeni (YEM). Sourced from local TV series, this dataset provides a realistic test for dialectal robustness.
Results and Discussion
The distilled models demonstrated a significant "distillation bonus," often outperforming their own teachers. This suggests that distillation, combined with Arabic-specific data, helps models specialize and ignore noise inherent in general-purpose multilingual models.
Key results include:
- Superior Performance: The best distilled model (DW-32-16++) achieved an overall 45.0% WER, beating the much larger SeamlessM4T-large-v2 (47.0%) and its teacher, Whisper-large-v2 (55.1%).
- Dialectal Robustness: On the new in-house data, the distilled model averaged 56.9% WER, significantly outperforming all other tested models.
- Efficiency: These models are 25% to 75% smaller than their teachers, making them viable for resource-constrained environments.
Error analysis revealed that large models are prone to "hallucinating" common web phrases (e.g., "subscribe to the channel") during silences. Furthermore, the study found a clear trade-off between quality and quantity: filtering for high-quality pseudo-labels yields better results than using larger volumes of lower-quality data.
Conclusion
While this work significantly advances Arabic ASR, limitations remain, including geographic gaps (e.g., Moroccan or Sudanese varieties) and the theatrical bias of TV-sourced data. Future work will focus on expanding to more low-resource dialects and exploring performance in code-switching scenarios.
Summary updated