The MADAR Shared Task on Arabic Fine-Grained Dialect Identification

Houda Bouamor, Sabit Hassan, Nizar Habash

Abstract

Arabic is a macrolanguage characterized by significant dialectal variation. While previous research has focused on broad regional classification, the Multi-Arabic Dialect Applications and Resources (MADAR) Shared Task introduces a benchmark for fine-grained identification at the city and country levels. This paper summarizes the results of two subtasks involving a travel-domain corpus and a Twitter-based corpus, highlighting the performance of traditional machine learning versus deep learning in data-constrained environments.

Introduction

The linguistic landscape of the Arabic-speaking world is a complex tapestry of dialects that vary significantly in phonology, morphology, and syntax. Traditional Arabic Natural Language Processing (NLP) has often simplified this complexity by categorizing dialects into broad regional groups like "Levantine" or "Maghrebi." However, these categories often fail to capture the nuances required for localized applications such as sentiment analysis or machine translation. The MADAR Shared Task was established to push the boundaries of dialect identification toward more granular levels—specifically identifying 25 distinct city dialects and 21 country-level dialects.

Datasets and Task Design

The shared task utilized two primary datasets developed under the MADAR project. The first is the MADAR Travel Domain Corpus, a parallel dataset derived from the Basic Traveling Expression Corpus (BTEC). It features translations of the same underlying sentences into the dialects of 25 major cities across the Arab world, plus Modern Standard Arabic (MSA). This "apples-to-apples" comparison allows models to focus purely on dialectal variation rather than topical differences.

Geographic coverage of the MADAR project across regions, countries, and cities.

The second dataset is the MADAR Twitter Corpus, which consists of nearly 3,000 user profiles from 21 countries. Unlike many social media datasets that rely on automated keyword filtering, this corpus was manually annotated by native speakers who verified user origins through bios, location tags, and content analysis.

Linguistic Variation

The challenge of fine-grained identification is best illustrated by the diverse ways a single concept is expressed. For instance, the phrase "I want a child's sweater" varies significantly from "Baddi kanzat atfal" in Aleppo to "Aba sutrat atfal" in Jeddah. These subtle differences are the primary focus of the shared task.

Examples of the same sentence translated into various Arabic dialects and Modern Standard Arabic (MSA).

Methodology

Participating teams (21 in total) employed a variety of techniques. Traditional machine learning approaches, such as Multinomial Naive Bayes (MNB) and Support Vector Machines (SVM) using word and character n-grams, were common. More recent entries utilized deep learning architectures, including Convolutional Neural Networks (CNNs), Bidirectional LSTMs (BiLSTMs), and early iterations of BERT-based models. Many successful systems relied on ensembles of multiple classifiers and sophisticated feature engineering like TF-IDF weighting.

Key Findings

The results revealed a notable distinction between the two subtasks:

  • Subtask 1 (City-level): Traditional ML models often outperformed neural networks. This "data hunger" of deep learning models became a bottleneck; with limited training data per specific city, simpler models with strong n-gram features proved more robust.
  • Subtask 2 (Country-level): Neural models excelled at handling the informal and noisy nature of social media data, suggesting that deep learning is better suited for high-volume, organic text.

Performance reached a macro-averaged F1 score of 67.32% for city identification (Team ArbDialectID) and 71.70% for country identification on Twitter (Team UBC-NLP). These results demonstrate that while fine-grained identification is significantly more difficult than regional classification, specialized architectures can achieve high accuracy.

Conclusion and Future Work

The MADAR Shared Task has established a new standard for Arabic dialect identification. Future efforts will focus on incorporating more "naturally occurring" speech beyond commissioned translations and addressing the "vanishing tweet" problem—the decay of social media datasets over time as accounts are deleted or made private. The ongoing goal is to create more stable and representative benchmarks for the evolving Arabic linguistic landscape.

Summary updated

BibTeX
@inproceedings{bouamor2019madar,
  author = {Houda Bouamor and Sabit Hassan and Nizar Habash},
  title = {{The MADAR Shared Task on Arabic Fine-Grained Dialect Identification}},
  booktitle = {WANLP @ ACL},
  year = {2019},
  doi = {10.18653/v1/w19-4622},
}
Figure