Arabic Train at NADI 2024 shared task: LLMs’ Ability to Translate Arabic Dialects into Modern Standard Arabic

Anastasiia Demidova, Hanin Atwany, Nour Rabih, Sanad Sha’banACL 2024

TL;DR

This paper investigates the performance of large language models (LLMs) like Jais, GPT-3.5, and GPT-4 on the challenging task of translating various Arabic dialects into Modern Standard Arabic (MSA). The study reveals that Jais, a model specifically pre-trained on Arabic data, significantly outperforms general-purpose LLMs, highlighting the importance of domain-specific pre-training for nuanced language tasks.

What Problem Does This Paper Address?

Machine translation (MT) systems face a significant hurdle when dealing with the diverse landscape of Arabic dialects. Unlike Modern Standard Arabic (MSA), which serves as the standardized written form, spoken dialects vary considerably, making direct translation difficult. This linguistic diversity hinders seamless communication and access to a vast amount of Arabic content. The NADI 2024 shared task, specifically Subtask 3, aims to address this by focusing on sentence-level translation from four major Arabic dialects (Egyptian, Emirati, Jordanian, and Palestinian) into MSA. This paper explores how contemporary LLMs handle this complex translation task.

Methods

The researchers evaluated several LLMs, including Jais, GPT-3.5, and GPT-4, using both zero-shot and few-shot learning approaches. In the zero-shot setting, models were prompted to translate dialectal Arabic to MSA without any prior examples. The few-shot setting provided a small number of examples to guide the models. A novel "combination approach" was also tested for Jais, leveraging its pre-existing familiarity with certain dialects (Jordanian and Palestinian) by using zero-shot for those and one-shot for others (Emirati and Egyptian). Additionally, the NLLB model was fine-tuned using a custom dataset created through a "back-translation" process involving Google Translate and French as an intermediate language to introduce linguistic diversity.

Datasets

The study utilized datasets from the NADI 2024 shared task, including a validation set of 400 sentences (100 per dialect) for model tuning and evaluation, and a test set of 2,000 sentences (500 per dialect). A custom dataset was also created for fine-tuning, employing a back-translation method using Google Translate between Arabic, English, and French. The MADAR Arabic Dialect Corpus and Lexicon was also used for fine-tuning models.

Key Insights and Ideas

  • Domain-specific pre-training is crucial: Jais, which was pre-trained specifically on Arabic data, significantly outperformed general-purpose LLMs like GPT-3.5 and GPT-4. This suggests that for highly nuanced linguistic tasks like dialect translation, specialized models offer a distinct advantage.
  • Few-shot learning improves performance: While zero-shot performance was evaluated, few-shot settings generally yielded better results, demonstrating the benefit of providing models with contextual examples.
  • Back-translation can enrich datasets: The custom dataset created using back-translation, particularly with French as an intermediate language, aimed to introduce linguistic diversity and potentially improve model robustness.
  • Named entities and cultural nuances are translation challenges: Error analysis revealed that named entities and culturally specific terms were common sources of translation errors, highlighting the difficulty of capturing the full meaning and context.

Key Results

On the validation dataset, Jais with the combination approach achieved the highest average BLEU score of 19.48. NLLB showed the least favorable performance with a BLEU score of 8.77. GPT-3.5 and GPT-4 achieved respectable scores, with GPT-3.5 (2-shot) scoring 16.67 and GPT-4 (0-shot) scoring 16.30. On the test dataset, Jais with the combination approach achieved the highest average BLEU score of 20.44. The paper also notes that Jais secured second place in the overall competition.

Limitations and Future Work

The authors acknowledge that while GPT-3.5 and GPT-4 are powerful, they are general-purpose models and not specifically optimized for Arabic dialects, which likely contributes to their lower performance compared to Jais. The study did not include other Arabic-specific models due to the focus on widely-used, general-purpose models for benchmarking. Future work could involve incorporating additional Arabic-specific models for a more comprehensive comparison and further exploring alternative methodologies to address the challenges of dialectal translation. The paper also suggests that reframing Automatic Arabic Dialect Identification (ADI) as a multi-label classification task could improve system accuracy.

Summary updated

BibTeX
@inproceedings{demidova2024arabic,
  author = {Anastasiia Demidova and Hanin Atwany and Nour Rabih and Sanad Sha’ban},
  title = {{Arabic Train at NADI 2024 shared task: LLMs’ Ability to Translate Arabic Dialects into Modern Standard Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure