TL;DR
This paper introduces TounsiBench, the first benchmark for evaluating Large Language Models (LLMs) on Tunisian Arabic. They found that most LLMs struggle with this dialect, highlighting a significant gap in current NLP capabilities for underrepresented languages. GPT-4o emerged as the most reliable LLM judge for this task.
What Problem Does This Paper Address?
The paper tackles the challenge of evaluating and improving the performance of Large Language Models (LLMs) for Tunisian Arabic, a dialect spoken by over 12 million people. Unlike Modern Standard Arabic (MSA), Tunisian Arabic has unique phonetic, lexical, and syntactic features, influenced by other languages, and lacks a standardized writing system. This linguistic complexity makes it difficult for NLP models, which are often trained on MSA or more widely resourced dialects, to understand and generate accurate, contextually appropriate responses. Existing research in Arabic NLP has largely overlooked Tunisian Arabic, leaving a critical gap in its evaluation and development.
Methods
The authors developed a novel benchmark, TounsiBench, by creating a dataset of 744 Tunisian Arabic instructions. This dataset was compiled through two methods: crowdsourcing naturally occurring chatbot queries from native speakers and translating a subset of the HuggingFace Helpful Instructions dataset. To evaluate LLMs, they employed a two-pronged approach:
- Human Evaluation: Three native Tunisian Arabic speakers acted as annotators, evaluating LLM responses based on four criteria: quality, correctness, relevance, and Tunisian Arabic usage. They used a pairwise comparison methodology and Krippendorff's alpha to measure inter-annotator agreement, refining their process through adjudication.
- LLM-as-a-Judge: They leveraged LLMs themselves to evaluate responses, using a standardized prompt adapted from the AlpacaEval benchmark. This approach aimed to provide a scalable and efficient evaluation method.
After evaluating ten LLMs, they used an Elo rating system to rank models based on human and LLM judgments. GPT-4o was selected as the primary judge due to its high agreement with human ratings.
Datasets
The primary dataset introduced in this paper is TounsiBench, comprising 744 unique Tunisian Arabic instructions. This dataset was created by:
- Crowdsourcing: Collecting 417 instructions from native Tunisian Arabic speakers.
- Translation: Translating 327 instructions from the HuggingFace Helpful Instructions dataset into Tunisian Arabic.
The dataset also includes gold-standard human-written responses for all 744 instructions, which are released to facilitate further research.
Key Insights and Ideas
- Tunisian Arabic is a significant challenge for current LLMs: The study reveals that most LLMs, even those claiming Arabic support, struggle to understand and generate responses in Tunisian Arabic, indicating a substantial gap in their capabilities for this dialect.
- LLM-as-a-Judge is a viable, but nuanced, evaluation method: While LLMs can effectively rank models, their judgments, particularly on dialectal adherence, can differ from human annotators. GPT-4o showed the highest agreement with human judges, making it a strong candidate for automated evaluation.
- Dialectal adherence is a critical, often overlooked, evaluation criterion: The study emphasizes the importance of evaluating not just factual correctness or fluency, but also how well a model adheres to the specific linguistic nuances of a dialect like Tunisian Arabic.
- LLMs may exhibit self-preference bias: The research suggests that LLMs tend to favor their own responses more than human judges do, a phenomenon that needs to be considered in evaluation.
Key Results
- GPT-4o leads the pack: GPT-4o was identified as the top-performing LLM in terms of overall quality and showed the highest agreement with human judges.
- Significant performance gaps: Most LLMs demonstrated considerable difficulty in recognizing and correctly responding in Tunisian Arabic, with many producing responses in MSA or other dialects.
- Human vs. LLM agreement on Tunisian Usage: While LLMs and humans largely agreed on the ranking of models for Tunisian usage proficiency, LLMs tended to overestimate the proficiency of lower-ranked models.
- Leaderboard established: The paper provides a final leaderboard of ten LLMs based on their performance across quality, correctness, relevance, and Tunisian Arabic usage, with GPT-4o at the top.
Limitations and Future Work
The authors acknowledge several limitations:
- Human evaluation bias: Human annotators might introduce subjective biases, such as a preference for shorter or longer responses.
- Script limitation: The benchmark focuses on Tunisian Arabic written in Arabic script, excluding the common use of Latin script or mixed scripts in digital communication.
- Dataset size: While diverse, the 744 instructions may not cover the full spectrum of queries and conversational contexts.
Future work could involve:
- Expanding the dataset with more instructions and diverse scripts.
- Fine-tuning LLMs on Tunisian Arabic data to improve their performance.
- Developing more robust evaluation methods that account for dialectal variations and script usage.
- Addressing potential biases and ethical considerations in LLM outputs for underrepresented dialects.
Summary updated