ADI-20: Arabic Dialect Identification dataset and models

Haroun Elleuch, Salima Mdhaffar, Yannick Estève, Fethi BougaresInterspeech 2025

TL;DR

This paper introduces ADI-20, a significantly expanded Arabic Dialect Identification (ADI) dataset covering 19 dialects plus Modern Standard Arabic (MSA), addressing the limitations of previous datasets. They also present high-performing ADI models based on ECAPA-TDNN and Whisper, demonstrating that effective dialect identification is achievable even with limited training data.

What Problem Does This Paper Address?

Accurately identifying spoken Arabic dialects is crucial for various natural language processing applications like machine translation and automatic speech recognition. However, Arabic dialects are incredibly diverse, with significant variations in pronunciation, vocabulary, and grammar, making this a challenging task. Existing datasets, like ADI-17, suffered from imbalanced dialect representation and incomplete coverage. This paper aims to tackle these issues by creating a more comprehensive dataset and exploring robust ADI models that can perform well even with less training data.

Methods

The authors leverage two primary model architectures:

  • ECAPA-TDNN: This is a well-established architecture for speaker recognition and has shown state-of-the-art performance in ADI tasks. The paper uses pre-trained ECAPA-TDNN models and fine-tunes them.
  • Whisper Encoder: They utilize the encoder part of OpenAI's Whisper model, a powerful, weakly supervised transformer trained on a massive amount of diverse audio data. The decoder is replaced with an attention pooling mechanism and a classification layer for ADI. They explore different sizes of Whisper models (small, medium, large) to investigate model complexity.

To assess the impact of data quantity, they create subsets of the ADI-17 dataset with reduced training hours per dialect (10h, 25h, 53h). They also experiment with data augmentation and layer freezing techniques for further performance improvements.

Datasets

  • ADI-17: The existing dataset used as a baseline, containing 3,033 hours of speech across 17 Arabic dialects. It's noted for its imbalanced dialect distribution.
  • ADI-20: The newly introduced dataset, expanding ADI-17 to include Modern Standard Arabic (MSA) and two additional dialects (Tunisian and Bahraini), bringing the total to 19 dialects plus MSA. It comprises 3,556 hours of speech, with a minimum of 53 hours per dialect.
  • ADI-5: Used for MSA evaluation sets.
  • TunSwitch: A dataset used for Tunisian dialect data.
  • Casablanca Dataset: Used for zero-shot evaluation to test model robustness on unseen data (TV dramas).

Key Insights and Ideas

  • Data is King (but not always): While more data generally leads to better performance, the paper shows that even with significantly reduced training data (e.g., 30% of ADI-17), performance doesn't drop drastically, especially with larger models like Whisper-large. This is a crucial finding for low-resource scenarios.
  • Whisper's Potential: The Whisper encoder, even without extensive fine-tuning on specific ADI tasks, shows strong performance, especially larger variants. Its broad pre-training on diverse audio seems to provide a good foundation.
  • ECAPA-TDNN's Edge (with caveats): ECAPA-TDNN, when compared to a similarly sized Whisper model, achieved a slightly better F1-score on ADI-17. The authors hypothesize this is due to its pre-training on speaker recognition, which shares some similarities with dialect identification.
  • Data Augmentation and Freezing Matter: For the ADI-20 dataset, techniques like data augmentation and freezing lower layers of the Whisper encoder significantly boost performance, especially on challenging unseen data like TV dramas.

Key Results

  • ADI-20 Dataset: Provides comprehensive coverage of 19 Arabic dialects plus MSA, with at least 53 hours of training data per dialect, totaling 3,556 hours.
  • Model Performance:
    • ECAPA-TDNN achieved a weighted F1-score of 93.16% on ADI-17 test sets.
  • Whisper-large, trained on the full ADI-17 dataset, achieved a weighted F1-score of 95.66% on ADI-17 test sets.
  • Using only 30% of the ADI-17 training data (ADI-17-53h), Whisper-small achieved a competitive F1-score of 94.42%, even outperforming the full ADI-17 training set in some cases.
  • On the Casablanca dataset (zero-shot), Whisper-large trained with ADI-20-53h+frz+aug achieved a significant F1-score of 62.74%, outperforming previous benchmarks by a large margin.
  • Error Analysis: Frequent misclassifications occur between geographically or linguistically close dialects (e.g., Jordanian as Lebanese/Palestinian/Syrian) and between MSA and North African dialects.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that obtaining audio data for dialects from countries like Somalia and Eritrea remains challenging and was not included in ADI-20.
  • While Whisper-large showed strong performance, the paper notes that ECAPA-TDNN models were not explored as extensively due to time constraints.
  • Future Work:
    • Further investigate ECAPA-TDNN models for ADI.
  • Extend research to city-level Arabic dialect identification to capture finer linguistic variations within countries.
  • Continue to expand dialectal coverage in future dataset versions.

Summary updated

BibTeX
@inproceedings{elleuch2025adi,
  author = {Haroun Elleuch and Salima Mdhaffar and Yannick Estève and Fethi Bougares},
  title = {{ADI-20: Arabic Dialect Identification dataset and models}},
  booktitle = {Interspeech},
  year = {2025},
}
Figure