Abstract
Identifying specific Arabic dialects from social media text is a complex task due to linguistic overlap and significant data imbalance. This study explores a two-stage BERT fine-tuning approach for the NADI Shared Task, achieving 5th place by adapting a pre-trained model to the informal domain of Twitter before task-specific training. The results demonstrate that domain-specific adaptation significantly outperforms traditional machine learning ensembles.
Introduction
Arabic is a pluricentric language where Modern Standard Arabic (MSA) serves formal purposes, while regional dialects dominate daily communication. Automatically identifying these dialects is essential for localized sentiment analysis and social media monitoring. However, the task is complicated by the granularity of 21 distinct country-level labels, high vocabulary overlap, and the noisy, informal nature of tweets. This paper addresses these challenges by comparing traditional machine learning with transformer-based deep learning architectures.
Datasets and the Imbalance Challenge
The research utilized the NADI Shared Task dataset, comprising 21,000 training tweets across 21 Arab countries. A significant hurdle identified by the researchers was the extreme data imbalance across the labels. For instance, Egypt accounted for over 21% of the data, whereas countries like Bahrain had negligible representation.
In addition to the labeled set, 10 million unlabeled tweets were leveraged for domain-specific language model adaptation to bridge the gap between formal pre-training data (like Wikipedia) and the informal nature of Twitter.
Methodology
The authors explored two distinct paths for classification:
1. Traditional Machine Learning Ensemble
They developed a baseline voting classifier that combined Linear and Non-linear SVMs (using character n-grams and word unigrams) with Bernoulli Naive Bayes. This approach relied on TF-IDF weighting and specific preprocessing to handle the noise of social media text.
2. Two-Phase BERT Fine-Tuning
The winning approach involved a two-stage process:
- Phase 1: Domain Adaptation: The BERT model was "warmed up" on 10 million unlabeled tweets for 120 hours to learn the specific nuances, slang, and structure of Arabic Twitter.
- Phase 2: Task Adaptation: A custom classification head—including batch normalization, dropout, and a linear layer—was added to the BERT base to map the text to the 21 specific country labels.
Key Results
The BERT-based approach significantly outperformed traditional methods. Fine-tuning the language model on the Twitter domain provided a crucial performance boost, raising the F1-score from 18.1 (traditional ensemble) to 24.05 (fine-tuned BERT).
The team secured 5th place out of 18 participants in the NADI Shared Task. The results prove that transformer-based context awareness is superior to simple n-gram counting, provided the model is sufficiently adapted to the target domain.
Discussion and Limitations
The study highlighted a "Gulf Cluster Problem," where dialects from countries like Bahrain, Kuwait, and the UAE are linguistically similar, leading to model confusion. Furthermore, the model's performance was closely tied to data volume; it frequently defaulted to predicting "Saudi Arabia" for Gulf tweets and struggled with minority classes.
Future work may involve hierarchical classification—first identifying broad regions (e.g., Maghreb, Nile Basin) before specific countries—and implementing specialized loss functions to address class imbalance more effectively.
Summary updated