ABDUL: A New Approach to Build Language Models for Dialects Using Formal Language Corpora Only

Yassine Toughrai, Kamel Smaïli, David LangloisNAACL 2025

TL;DR

This paper introduces ABDUL, a novel approach to building language models for Arabic dialects by using a phoneme-like transcription of formal Arabic. This method bypasses the need for scarce dialect-specific data, achieving competitive or superior performance in tasks like emotion classification and named entity recognition across various Arabic dialects.

What Problem Does This Paper Address?

Natural Language Processing (NLP) for Arabic dialects faces significant hurdles due to their diglossic nature, phonetic variability, and a severe lack of readily available, high-quality resources. Unlike Modern Standard Arabic (MSA), which has formal rules and extensive corpora, dialects are often spoken, transcribed inconsistently, and heavily influenced by loanwords from languages like French and Spanish. This makes it challenging to train robust NLP models that can understand and process these variations effectively. Existing approaches often rely on dialect-specific datasets, which are difficult to obtain and curate, or use multilingual models that may not capture the nuances of individual dialects.

Methods

The core of ABDUL's innovation lies in its preprocessing pipeline. It transforms Arabic script into a phoneme-like Latin representation. This involves:

  • Dediacritization: Removing short vowels and diacritics, as consonants are considered to carry the fundamental semantic meaning in Arabic.
  • Long Vowel Retention: Preserving long vowels to maintain essential phonetic cues and reduce ambiguity.
  • Simplified Transliteration: Adapting a system inspired by Buckwalter's transliteration, this step clusters phonetically similar sounds to improve alignment with dialectal patterns.

This phoneme-like representation allows for cross-script and cross-dialect generalization, enabling models trained on formal Arabic data to perform well on dialects. The authors then pretrain a BERT model from scratch using this processed data.

Datasets

  • Pretraining: The Arabic subset of the OSCAR corpus, a large collection of web-sourced text representing MSA.
  • Emotion Classification: SemEval 2025 Task 11-A dataset, focusing on Moroccan and Algerian Arabic.
  • Named Entity Recognition (NER): WikiFANE (MSA and NADs), DzNER (Algerian Arabic), and DarNER (Moroccan Arabic).

Key Insights and Ideas

  • Phonetic Similarity Bridges Dialects: The key insight is that Arabic dialects are highly phonetic, and by representing them in a standardized, phoneme-like way, we can leverage the vast resources of formal Arabic.
  • Consonants as Anchors: Focusing on consonants and long vowels, while simplifying short vowels, aligns with the principle that consonants carry core meaning and helps normalize phonetic variations.
  • Formal Data for Dialectal Tasks: The success of ABDUL demonstrates that it's possible to achieve strong performance on low-resource dialects using only formal language corpora, significantly reducing the dependency on scarce dialect-specific data.
  • Vocabulary Efficiency: ABDUL achieves comparable or better performance than models with much larger vocabularies, highlighting the efficiency gained from its preprocessing.

Key Results

  • Emotion Classification: ABDUL achieves a macro-F1 score of 0.5315 for Algerian Arabic, ranking second behind DziriBERT. For Moroccan Arabic, it achieves 0.4519, closely matching AraBERT. Across all dialects, ABDUL achieves an average macro-F1 of 0.4915, outperforming DarijaBERT and DziriBERT.
  • Named Entity Recognition (NER): For MSA, ABDUL achieves an F1 score of 0.4646, on par with mBERT. For Algerian Arabic, it achieves the highest F1 score of 0.6828, significantly outperforming DziriBERT. For Moroccan Arabic, ABDUL attains an F1 score of 0.6557. Across all variants, ABDUL achieves an overall F1 score of 0.6010, outperforming DarijaBERT and DziriBERT.
  • Vocabulary Size: Notably, ABDUL uses a vocabulary size of 30,522 tokens, which is significantly smaller than many of the benchmark models (e.g., mBERT with 119,547 tokens), yet it achieves competitive results.

Limitations and Future Work

  • Latin-Script Arabizi: The current approach does not support Latin-script Arabizi dialects, which are prevalent in informal online communication.
  • Broader NLP Tasks: The authors plan to expand ABDUL's applicability to tasks like machine translation and text generation.
  • LLM for Arabic Dialects: A long-term goal is to build a large language model (LLM) specifically for Arabic dialects.
  • Impact of Pretraining Languages: Further investigation into how different formal languages used in pretraining (e.g., French, Spanish) influence model generalization across dialects is suggested.
  • Architecture Exploration: The paper acknowledges that results are "low for state-of-the-art models" and suggests exploring other architectures to improve performance.

Summary updated

BibTeX
@inproceedings{toughrai2025abdul,
  author = {Yassine Toughrai and Kamel Smaïli and David Langlois},
  title = {{ABDUL: A New Approach to Build Language Models for Dialects Using Formal Language Corpora Only}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure