TL;DR
This study investigates whether prompting Large Language Models (LLMs) in a task's native language (Arabic) is more effective than using English. Surprisingly, across 11 diverse NLP tasks and three major models, English-based "non-native" prompts consistently outperformed native ones, revealing a persistent English-centric bias even in models specifically designed for Arabic.
What Problem Does This Paper Address?
LLMs are predominantly trained on English-heavy datasets. While they demonstrate impressive multilingual capabilities, a critical question remains: What is the best language to use for the prompt itself?
If you are asking a model to classify Arabic tweets for hate speech, should the instructions and labels be in Arabic (Native), English (Non-native), or a mix of both? While intuition suggests that a "native" approach would be more natural, the underlying training distribution of these models often favors English. This paper quantifies this gap for medium-to-low-resourced languages, specifically focusing on the Arabic social and news media context.
Methods
The researchers conducted 198 experimental setups comparing three prompting strategies across three LLMs: GPT-4o, Llama-3.1-8b-Instruct, and Jais-13b-chat (an Arabic-centric model).
The three prompt formulations were:
- Native: Arabic instructions + Arabic input + Arabic labels.
- Non-native: English instructions + Arabic input + English labels.
- Mixed: Arabic instructions + Arabic input + English labels.
The study evaluated these strategies in both zero-shot and 3-shot settings. For the few-shot experiments, they used the Maximal Marginal Relevance (MMR) method to select examples that were both relevant to the input and diverse.
Datasets
The study utilized 11 different Arabic datasets covering a wide range of social media and news analysis tasks:
- Safety/Harm: Hate Speech (OSACT2020), Adult Content (ASAD), Harmful Content (CT-CWT-22), Offensive Language (OffensEval2020).
- Information Integrity: Factuality (ANS), Check-worthiness (CT-CWT-22), Claim Detection (CT-CWT-22), Propaganda Detection (WANLP22).
- Content Analysis: Spam Detection (ASAD), Subjectivity Identification (ThatiAR), Attention-worthiness (CT-CWT-22).
To manage costs and computation, the authors created a sampled test set of 8,758 total data points (roughly 1,000 per task where possible).
Key Insights and Ideas
- English is the "Internal Language" of LLMs: Across almost all tasks and models, non-native (English) prompts performed the best. This suggests that the models' internal reasoning and instruction-following capabilities are most robust when triggered by English.
- The Jais Paradox: Even Jais-13b-chat, a model specifically designed and trained for Arabic, performed worst with native Arabic prompts. The authors attribute this to the fact that a large portion of Jais's instruction-tuning data was translated from English, causing it to inherit English-centric behaviors.
- Mixed Prompts are a Strong Few-Shot Contender: In the 3-shot setup, Llama-3.1-8b-Instruct performed 13% better with mixed prompts than with either pure native or non-native prompts. This indicates that providing Arabic context while using English labels can help "anchor" the model's classification performance.
- Instruction-Following Fragility: The Arabic-centric Jais model struggled significantly with output formatting, often including unwanted explanations or failing to return only the requested label, unlike GPT-4o and Llama which were much more disciplined.
Key Results
The quantitative results show a clear hierarchy of performance: Non-native > Mixed > Native.
- Zero-Shot: Non-native prompts were the clear winner across all models. GPT-4o showed the most robustness, with only minimal performance drops between prompt types, while Jais saw a significant boost when switched to English instructions.
- Few-Shot (3-shot): Performance improved across the board compared to zero-shot. GPT-4o remained the top performer, achieving its highest scores with non-native prompts.
Despite these improvements, the authors note that LLMs still generally lag behind the current State-of-the-Art (SOTA) results achieved by specialized, supervised models.
Limitations and Future Work
- Model Hallucinations: Jais frequently hallucinated or refused to predict (e.g., "it goes against our use case policy") when prompted in Arabic.
- Safety Filters: GPT-4o occasionally triggered "ResponsibleAIPolicyViolation" errors on certain sensitive social media inputs, requiring random label assignment to complete the evaluation.
- Retrieval Quality: The inconsistency in few-shot performance for some tasks was attributed to the relevance of the retrieved examples; text similarity does not always guarantee label similarity.
Future Work: The authors emphasize the need for fine-tuning LLMs on high-quality, native-language instruction-following datasets to bridge the performance gap and make models truly "native" in their reasoning. All prompts and configurations will be released via the LLMeBench framework.
Summary updated