To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation

Abdul Waheed, Karima Kadaoui, Muhammad Abdul-Mageed

Abstract

Arabic is known to present unique challenges for Automatic Speech Recognition (ASR). On one hand, its rich linguistic diversity and wide range of dialects complicate the development of robust, inclusive models. On the other, current multilingual ASR models are compute-intensive and lack proper comprehensive evaluations. In light of these challenges, we distill knowledge from large teacher models into smaller student variants that are more efficient. We also introduce a novel human-annotated dataset covering five under-represented Arabic dialects for evaluation. We further evaluate both our models and existing SoTA multilingual models on both standard available benchmarks and our new dialectal data. Our best-distilled model’s overall performance (45.0% WER) surpasses that of a SoTA model twice its size (SeamlessM4T-large-v2, WER=47.0%) and its teacher model (Whisper-large-v2, WER=55.1%), and its average performance on our new dialectal data (56.9% WER) outperforms all other models. To gain more insight into the poor performance of these models on dialectal data, we conduct an error analysis and report the main types of errors the different models tend to make.

Introduction

Automatic Speech Recognition (ASR) for Arabic is a linguistic minefield due to diglossia, where Modern Standard Arabic (MSA) is used for formal writing while distinct regional dialects (Dialectal Arabic or DA) are used in daily life. Current State-of-the-Art (SoTA) multilingual models like Whisper and SeamlessM4T are "compute-heavy" giants that, while proficient in MSA, often struggle with the nuances of regional dialects. Furthermore, existing benchmarks are heavily biased toward MSA, leaving a gap in our understanding of how these models perform for speakers in regions like Algiers or Sana'a. This research addresses two main gaps: the lack of efficient models and the absence of comprehensive evaluation data for under-represented Arabic dialects.

Methodology

The authors employ Knowledge Distillation (KD), where a large, pre-trained "teacher" model (Whisper-large-v2) trains a smaller "student" model. The process involves:

  1. Pseudo-Labeling: Using the teacher to transcribe massive amounts of unlabeled Arabic audio.
  2. Filtering: Applying a Word Error Rate (WER) threshold to filter out low-quality transcriptions, ensuring the student doesn't learn the teacher's mistakes.
  3. Architecture: Initializing student models using "maximally spaced" layers from the teacher to retain structural knowledge.
  4. Loss Function: Combining standard transcription loss with Kullback-Leibler (KL) divergence to force the student to mimic the teacher's probability distribution.

Data and Evaluation

The study utilizes standard benchmarks (Common Voice, FLEURS, MGB) alongside a crucial new contribution: a human-annotated "In-House" dataset covering five under-represented dialects: Algerian (ALG), Jordanian (JOR), Palestinian (PAL), Emirati (UAE), and Yemeni (YEM). Sourced from local TV series, this dataset provides a realistic test for dialectal robustness.

Statistics of the novel human-annotated in-house dialectal dataset covering five under-represented Arabic dialects.

Results and Discussion

The distilled models demonstrated a significant "distillation bonus," often outperforming their own teachers. This suggests that distillation, combined with Arabic-specific data, helps models specialize and ignore noise inherent in general-purpose multilingual models.

Key results include:

  • Superior Performance: The best distilled model (DW-32-16++) achieved an overall 45.0% WER, beating the much larger SeamlessM4T-large-v2 (47.0%) and its teacher, Whisper-large-v2 (55.1%).
  • Dialectal Robustness: On the new in-house data, the distilled model averaged 56.9% WER, significantly outperforming all other tested models.
  • Efficiency: These models are 25% to 75% smaller than their teachers, making them viable for resource-constrained environments.
Comprehensive comparison of Word Error Rate (WER) across standard benchmarks and the new in-house dialectal dataset, highlighting the performance of distilled models.

Error analysis revealed that large models are prone to "hallucinating" common web phrases (e.g., "subscribe to the channel") during silences. Furthermore, the study found a clear trade-off between quality and quantity: filtering for high-quality pseudo-labels yields better results than using larger volumes of lower-quality data.

Conclusion

While this work significantly advances Arabic ASR, limitations remain, including geographic gaps (e.g., Moroccan or Sudanese varieties) and the theatrical bias of TV-sourced data. Future work will focus on expanding to more low-resource dialects and exploring performance in code-switching scenarios.

Summary updated

Abstract

Arabic is known to present unique challengesfor Automatic Speech Recognition (ASR). Onone hand, its rich linguistic diversity andwide range of dialects complicate the de-velopment of robust, inclusive models. Onthe other, current multilingual ASR modelsare compute-intensive and lack proper com-prehensive evaluations. In light of thesechallenges, we distill knowledge from largeteacher models into smaller student variantsthat more efficient. We also introduce a novelhuman-annotated dataset covering five under-represented Arabic dialects for evaluation. Wefurther evaluate both our models and existingSoTA multilingual models on both standardavailable benchmarks and our new dialectaldata. Our best-distilled model’s overall perfor-mance (45.0% WER) surpasses that of a SoTAmodel twice its size (SeamlessM4T-large-v2,WER=47.0%) and its teacher model (Whisper-large-v2, WER=55.1%), and its average perfor-mance on our new dialectal data (56.9% WER)outperforms all other models. To gain more in-sight into the poor performance of these modelson dialectal data, we conduct an error analysisand report the main types of errors the differentmodels tend to make. The GitHub repositoryfor the project is available at https://github.com/UBC-NLP/distill-whisper-ar.
BibTeX
@inproceedings{waheed2024distill,
  author = {Abdul Waheed and Karima Kadaoui and Muhammad Abdul-Mageed},
  title = {{To Distill or Not to Distill? On the Robustness of Robust Knowledge Distillation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=avP7MPZrXz},
}
Figure