Alson at NADI 2024 shared task: Alson - A fine-tuned model for Arabic Dialect Translation

Manan AlMusallam, Samar AhmadACL 2024

TL;DR

This paper presents "Alson," a fine-tuned AraT5v2 model for translating four Arabic dialects (Egyptian, Emirati, Jordanian, Palestinian) into Modern Standard Arabic (MSA). The model achieved the best performance in the NADI 2024 shared task, highlighting the effectiveness of large pre-trained models and carefully curated datasets for this challenging translation problem.

What Problem Does This Paper Address?

The Arabic language presents a unique challenge for machine translation due to its rich tapestry of dialects (DA) coexisting with Modern Standard Arabic (MSA). While MSA serves as the formal written and broadcast standard, everyday communication heavily relies on diverse, informal dialects. This disparity creates a significant barrier for effective cross-dialect communication and information access. Developing accurate DA-MSA translation systems is crucial but hampered by the limited availability of parallel datasets compared to widely studied languages like English. This paper tackles the specific problem of translating four major Arabic dialects into MSA, aiming to improve the accuracy and accessibility of Arabic NLP tools.

Methods

The core of the "Alson" system is the AraT5v2 model, a pre-trained transformer specifically designed for Arabic text generation tasks. The authors fine-tuned three variants of this model: AraT5v2-msa-small, AraT5v2-msa-base, and AraT5v2-base-1024. The fine-tuning process involved adapting these models to the specific task of dialect-to-MSA translation. They also explored the use of ChatGPT (GPT-3.5) as a zero-shot translator for Jordanian and Palestinian dialects, demonstrating the potential of large language models without explicit fine-tuning. Data preprocessing involved standard techniques like removing URLs, punctuation, digits, and extra whitespace to ensure clean input for the models.

Datasets

The research utilized several datasets:

  • MADAR Arabic dialect corpus and lexicon: A foundational dataset covering 25 Arabic-speaking cities, including dialectal and MSA data.
  • Emi-NADI corpus: Specifically for the Emirati dialect, addressing a gap in the MADAR corpus.
  • NADI2021 dataset: Augmented for Jordanian and Palestinian dialects, providing additional parallel data.
  • Augmented Dataset (4 dialects + Jor-Pla-NADI): This combined dataset, used for training, comprised data from MADAR, Emi-NADI, and the augmented Jordanian and Palestinian data.
  • Development and Test Sets: Provided by the NADI 2024 shared task, with specific splits for evaluation.

Key Insights and Ideas

  • Pre-trained models are powerful: Fine-tuning large, pre-trained Arabic models like AraT5v2 is highly effective for dialect translation, outperforming simpler models.
  • Dataset composition matters: The combined dataset of four dialects (4 dialects + Jor-Pla-NADI) consistently yielded better results than the broader MADAR corpus. This suggests that focusing on specific dialects can lead to more effective learning.
  • Dialect similarity influences performance: The high accuracy for Jordanian and Palestinian dialects is attributed to their similarity to each other and to MSA, facilitating easier translation. Conversely, the Emirati dialect showed lower accuracy, potentially due to greater divergence.
  • ChatGPT shows promise: While not fine-tuned, ChatGPT demonstrated capabilities in translating dialects, indicating its potential as a baseline or supplementary tool.
  • Hyperparameter tuning is nuanced: While extensive tuning was performed, significant performance gains were marginal. The authors found that increasing epochs beyond a certain point could even decrease accuracy, suggesting a need for careful optimization rather than brute-force iteration.

Key Results

The AraT5v2-base-1024 model, trained on the 4 dialects + Jor-Pla-NADI dataset, achieved the best performance.

  • Development Set: A BLEU score of 0.1650.
  • Test Set: A BLEU score of 0.1746.

The model also showed varying performance across individual dialects, with Egyptian and Emirati dialects posing more challenges than Jordanian and Palestinian.

Limitations and Future Work

The authors acknowledge several limitations:

  • Resource Constraints: Experiments were limited by the computational resources available on Google Colab Pro, preventing larger-scale testing and more extensive hyperparameter tuning.
  • RAM Limitations: Occasional crashes during model testing due to RAM issues required workarounds like segmenting the test file.
  • Dialect-Specific Challenges: The model struggled with dialect-specific words and culturally nuanced expressions, particularly in Egyptian and Emirati dialects, due to phonetic variations and divergence from MSA.

Future work includes:

  • Expanding Dialect Coverage: Incorporating more Arabic dialects using APIs like GPT or specialized large language models.
  • Volunteer Reviewers: Engaging human reviewers to assess translation quality more comprehensively.
  • Practical Applications: Exploring real-world applications of the developed translation system.
  • Addressing Resource Limitations: Investigating the use of larger servers with advanced resources to overcome current constraints.

Summary updated

BibTeX
@inproceedings{almusallam2024alson,
  author = {Manan AlMusallam and Samar Ahmad},
  title = {{Alson at NADI 2024 shared task: Alson - A fine-tuned model for Arabic Dialect Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure