TL;DR
This paper benchmarks the LLaMA-3-70B model on a comprehensive set of 20 Arabic Natural Language Generation (NLG) tasks, revealing that it lags behind closed-source models like ChatGPT and even smaller, specialized Arabic models. The findings highlight the ongoing challenges in achieving high performance for Arabic NLG tasks with general-purpose LLMs and underscore the need for more Arabic-focused model development.
What Problem Does This Paper Address?
The rapid advancement of open-source Large Language Models (LLMs) like LLaMA-3 has been impressive, but their performance in non-English languages, particularly those with rich morphology and diverse dialects like Arabic, remains underexplored. While closed-source LLMs like ChatGPT have shown strong capabilities, their proprietary nature limits research. This paper aims to bridge this gap by providing a thorough evaluation of the state-of-the-art open-source LLM, LLaMA-3-70B, on a wide array of Arabic NLG tasks, comparing it against both closed-source models and specialized Arabic models.
Methods
The researchers conducted a systematic evaluation of LLaMA-3-70B across 20 distinct Arabic NLG tasks, categorized into 10 clusters. They employed zero-shot, 3-shot, and 5-shot learning settings to assess the model's performance. The evaluation involved comparing LLaMA-3-70B's outputs against those of ChatGPT and a fine-tuned Arabic model called AraT5. Prompt design was standardized to ensure fair comparison across models, with a focus on role assignment, task description, expected outcome, and k-shot examples.
Datasets
The evaluation utilized 20 distinct Arabic NLG tasks compiled from the Dolphin benchmark (Nagoudi et al., 2023). These tasks cover a range of NLG capabilities, including:
- Machine Translation: Evaluating translation between various languages and Arabic, including Modern Standard Arabic (MSA) and dialectal Arabic (DA).
- Code-Switching: Translating code-switched Arabic texts into foreign languages.
- Text Summarization (TS): Generating summaries for Arabic and multilingual news articles.
- News Title Generation (NTG): Crafting appropriate titles for news articles.
- Question Answering (QA): Including extractive, retrieval, open-domain, and multi-choice QA tasks.
- Question Generation (QG): Creating questions from provided passages.
- Paraphrase: Generating paraphrases of Arabic sentences.
- Transliteration: Converting text between writing systems while preserving pronunciation.
- Text Rewriting: Recreating text in a target style while maintaining original meaning.
- Diacritization: Restoring missing diacritics in Arabic texts.
The study also referenced datasets like UN parallel corpus, MultiUN corpus, Darija, NArabizi, MDP corpus, AraOPUS-20, MassiveSum, XLSum, Cross-Sum, ANT, MarSum, Arabic QA dataset ARCD, MLQA, XQUAD, TyDiQA, GoldP, LAREQA, AraQuAD-R, DAWQAS, EXAMS, AraPara, ASEP, APB, TaPaCo, ANETA, ATAR, NETransliteration, Dial2MSA, and Arabic parallel gender corpus.
Key Insights and Ideas
- LLaMA-3-70B's Arabic Gap: Despite being a leading open-source LLM, LLaMA-3-70B significantly underperforms compared to closed-source models like ChatGPT on Arabic NLG tasks. This suggests that general-purpose LLMs, even large ones, may struggle with the nuances of Arabic morphology and dialectal variations without specific Arabic-focused training.
- Specialized Models Shine: Smaller, fine-tuned Arabic models, particularly AraT5, consistently outperform both LLaMA-3-70B and ChatGPT across a majority of the evaluated tasks. This highlights the critical importance of domain-specific fine-tuning and specialized architectures for achieving high performance in low-resource languages.
- Dialectal Challenges Persist: The evaluation on dialectal Arabic machine translation tasks further emphasizes LLaMA-3-70B's inferiority compared to ChatGPT. While the performance gap narrows with more few-shot examples, it indicates that more dialectal training data is crucial for open LLMs to bridge this divide.
- Prompting's Role: The study acknowledges the influence of prompt design, noting that English prompts generally outperformed Arabic ones in initial experiments. A standardized English prompt template was developed to ensure consistency.
Key Results
- Overall Performance: ChatGPT generally outperforms LLaMA-3-70B on most NLG tasks.
- AraT5's Dominance: The fine-tuned AraT5 model achieves the best scores on the majority of tasks, significantly outperforming both LLaMA-3-70B and ChatGPT.
- Specific Task Performance:
- Text Rewriting: LLaMA-3-70B with 3-shots (41.83 BLEU) outperformed ChatGPT, but AraT5 was superior.
- Question Answering: LLaMA-3-70B with 3-shot achieved a score of 70.53, outperforming ChatGPT (54.14 at best), but both were surpassed by AraT5 (81.45).
- Machine Translation: ChatGPT outperformed LLaMA-3-70B on all evaluated X → Arabic MT tasks.
- Code-Switched Translation: LLaMA-3-70B (38.80 BLEU) with 5-shot slightly outperformed ChatGPT on the Jordanian Arabic task.
- Dialectal MT: ChatGPT consistently outperformed LLaMA-3-70B on MSA and all five evaluated Arabic dialects for English MT tasks.
Limitations and Future Work
- Model Variation: The study focused on LLaMA-3-70B and did not include other open LLMs like AceGPT-7B, Jais-13B, or Gemma-7B. The authors intend to include these in future work.
- ChatGPT Version: The evaluation used a snapshot of ChatGPT (gpt-3.5-turbo) from March 2023. As ChatGPT versions are updated, its performance can vary, meaning results should be interpreted with this in mind.
- Arabic Data Scarcity: The inferior performance of LLaMA-3-70B is suspected to be due to insufficient Arabic data during pre-training and a lack of Arabic instruction data for fine-tuning.
- Prompt Sensitivity: While a standardized prompt was used, different prompt designs might yield better results.
- Few-Shot Learning: The improvement with few-shot learning was not always consistent, suggesting it's model- and task-dependent.
Future work includes scaling the benchmark to other open LLMs and expanding to more Arabic language understanding tasks. The authors hope their findings will spur research into developing stronger, Arabic-focused LLMs.
Summary updated