Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling

Muhammad Khalifa, Muhammad Abdul-Mageed, Khaled Shaalan

Abstract

Arabic sequence labeling faces significant challenges due to diglossia, where Modern Standard Arabic (MSA) possesses abundant labeled resources while regional dialects (DA) remain resource-scarce. This paper introduces an iterative self-training framework that bridges this gap by leveraging unlabeled dialectal data. Our approach achieves a 10.03 F1 improvement in zero-shot Named Entity Recognition (NER) and establishes new state-of-the-art benchmarks for Part-of-Speech (POS) tagging across multiple dialects, demonstrating the effectiveness of domain-specific adaptation in low-resource settings.

Introduction

Arabic is characterized by diglossia: a formal standard (MSA) used in official contexts and various distinct dialects (e.g., Egyptian, Gulf, Levantine) used in daily communication. While NLP models perform well on MSA, their performance typically drops by up to 20 F1 points when applied to dialects in zero-shot scenarios. This discrepancy arises from differences in vocabulary, morphology, and syntax. This research explores how to utilize resource-rich MSA data to build high-performing models for resource-scarce dialects without requiring extensive manual labeling.

Methodology

The core of our approach is an iterative self-training loop built upon the XLM-RoBERTa (XLM-R) transformer architecture.

The iterative self-training pipeline for adapting MSA models to dialectal Arabic.

The process follows a four-stage cycle:

  1. Initial Fine-tuning: The model is first trained on labeled MSA data.
  2. Pseudo-labeling: The MSA-trained model predicts labels for a large corpus of unlabeled dialectal text from the Arabic Online Commentary (AOC) dataset.
  3. Selection: High-confidence predictions are selected using either a fixed confidence threshold (e.g., $\tau = 0.90$) or a fixed-size selection of the top $S$ examples.
  4. Augmentation: These pseudo-labeled examples are added to the training set, and the model is re-trained.

Key Insights

A critical finding is the "False Trigger" effect. In MSA, specific words often act as strong indicators for named entities (e.g., "Prophet" preceding a name). In dialects, these same words may function as common nouns or interjections. Self-training allows the model to adapt to these dialectal nuances, significantly reducing false positives.

Furthermore, the study demonstrates that domain-specific adaptation can be more impactful than model scale; a smaller "Base" model with self-training can outperform a "Large" model without it. This suggests that targeted learning from unlabeled data is a viable alternative to increasing parameter counts.

Experimental Results

The proposed method sets new state-of-the-art (SOTA) results for both NER and POS tagging. In zero-shot NER, self-training improved MSA-to-DA transfer by 10.03 F1 points. For POS tagging, accuracy increased across all dialects, with the Egyptian (EGY) dialect seeing a 2.41% gain.

Performance comparison between self-training and standard fine-tuning across different percentages of labeled dialectal data for EGY, GLF, and LEV dialects.

As shown in the figure above, self-training consistently outperforms standard fine-tuning across all tested dialects, even as the amount of manually labeled "gold" data increases. This confirms that self-training provides a robust benefit in both zero-shot and few-shot scenarios.

Conclusion

This work demonstrates that iterative self-training is a powerful tool for bridging the linguistic gap in multi-dialectal Arabic NLP. By effectively utilizing unlabeled data, models can overcome the limitations of MSA-centric training and adapt to the unique characteristics of regional dialects. Future work will explore applying this language-agnostic framework to other tasks such as sentiment analysis and machine translation.

Summary updated

BibTeX
@inproceedings{khalifa2021self,
  author = {Muhammad Khalifa and Muhammad Abdul-Mageed and Khaled Shaalan},
  title = {{Self-Training Pre-Trained Language Models for Zero- and Few-Shot Multi-Dialectal Arabic Sequence Labeling}},
  booktitle = {EACL},
  year = {2021},
  url = {https://openreview.net/forum?id=HYWbU87nPz9},
}
Figure