Native vs Non-Native Language Prompting: A Comparative Analysis

Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, Firoj AlamarXiv 2024 · arXiv

TL;DR

This study investigates whether prompting Large Language Models (LLMs) in a task's native language (Arabic) is more effective than using English. Surprisingly, across 11 diverse NLP tasks and three major models, English-based "non-native" prompts consistently outperformed native ones, revealing a persistent English-centric bias even in models specifically designed for Arabic.

What Problem Does This Paper Address?

LLMs are predominantly trained on English-heavy datasets. While they demonstrate impressive multilingual capabilities, a critical question remains: What is the best language to use for the prompt itself?

If you are asking a model to classify Arabic tweets for hate speech, should the instructions and labels be in Arabic (Native), English (Non-native), or a mix of both? While intuition suggests that a "native" approach would be more natural, the underlying training distribution of these models often favors English. This paper quantifies this gap for medium-to-low-resourced languages, specifically focusing on the Arabic social and news media context.

Methods

The researchers conducted 198 experimental setups comparing three prompting strategies across three LLMs: GPT-4o, Llama-3.1-8b-Instruct, and Jais-13b-chat (an Arabic-centric model).

The three prompt formulations were:

  1. Native: Arabic instructions + Arabic input + Arabic labels.
  2. Non-native: English instructions + Arabic input + English labels.
  3. Mixed: Arabic instructions + Arabic input + English labels.
Examples of the three prompting techniques: native, non-native, and mixed language.

The study evaluated these strategies in both zero-shot and 3-shot settings. For the few-shot experiments, they used the Maximal Marginal Relevance (MMR) method to select examples that were both relevant to the input and diverse.

Datasets

The study utilized 11 different Arabic datasets covering a wide range of social media and news analysis tasks:

  • Safety/Harm: Hate Speech (OSACT2020), Adult Content (ASAD), Harmful Content (CT-CWT-22), Offensive Language (OffensEval2020).
  • Information Integrity: Factuality (ANS), Check-worthiness (CT-CWT-22), Claim Detection (CT-CWT-22), Propaganda Detection (WANLP22).
  • Content Analysis: Spam Detection (ASAD), Subjectivity Identification (ThatiAR), Attention-worthiness (CT-CWT-22).

To manage costs and computation, the authors created a sampled test set of 8,758 total data points (roughly 1,000 per task where possible).

Table 1. Data distribution across various tasks and datasets. Test (Orig.): original test set. Test: sampled test sets.

Key Insights and Ideas

  • English is the "Internal Language" of LLMs: Across almost all tasks and models, non-native (English) prompts performed the best. This suggests that the models' internal reasoning and instruction-following capabilities are most robust when triggered by English.
  • The Jais Paradox: Even Jais-13b-chat, a model specifically designed and trained for Arabic, performed worst with native Arabic prompts. The authors attribute this to the fact that a large portion of Jais's instruction-tuning data was translated from English, causing it to inherit English-centric behaviors.
  • Mixed Prompts are a Strong Few-Shot Contender: In the 3-shot setup, Llama-3.1-8b-Instruct performed 13% better with mixed prompts than with either pure native or non-native prompts. This indicates that providing Arabic context while using English labels can help "anchor" the model's classification performance.
  • Instruction-Following Fragility: The Arabic-centric Jais model struggled significantly with output formatting, often including unwanted explanations or failing to return only the requested label, unlike GPT-4o and Llama which were much more disciplined.

Key Results

The quantitative results show a clear hierarchy of performance: Non-native > Mixed > Native.

  • Zero-Shot: Non-native prompts were the clear winner across all models. GPT-4o showed the most robustness, with only minimal performance drops between prompt types, while Jais saw a significant boost when switched to English instructions.
  • Few-Shot (3-shot): Performance improved across the board compared to zero-shot. GPT-4o remained the top performer, achieving its highest scores with non-native prompts.
Zero-shot average results across models and prompt types.
3-shot average results across models and prompt types.

Despite these improvements, the authors note that LLMs still generally lag behind the current State-of-the-Art (SOTA) results achieved by specialized, supervised models.

Limitations and Future Work

  • Model Hallucinations: Jais frequently hallucinated or refused to predict (e.g., "it goes against our use case policy") when prompted in Arabic.
  • Safety Filters: GPT-4o occasionally triggered "ResponsibleAIPolicyViolation" errors on certain sensitive social media inputs, requiring random label assignment to complete the evaluation.
  • Retrieval Quality: The inconsistency in few-shot performance for some tasks was attributed to the relevance of the retrieved examples; text similarity does not always guarantee label similarity.

Future Work: The authors emphasize the need for fine-tuning LLMs on high-quality, native-language instruction-following datasets to bridge the performance gap and make models truly "native" in their reasoning. All prompts and configurations will be released via the LLMeBench framework.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable abilities in different fields, including standard Natural Language Processing (NLP) tasks. To elicit knowledge from LLMs, prompts play a key role, consisting of natural language instructions. Most open and closed source LLMs are trained on available labeled and unlabeled resources--digital content such as text, images, audio, and videos. Hence, these models have better knowledge for high-resourced languages but struggle with low-resourced languages. Since prompts play a crucial role in understanding their capabilities, the language used for prompts remains an important research question. Although there has been significant research in this area, it is still limited, and less has been explored for medium to low-resourced languages. In this study, we investigate different prompting strategies (native vs. non-native) on 11 different NLP tasks associated with 12 different Arabic datasets (9.7K data points). In total, we conducted 197 experiments involving 3 LLMs, 12 datasets, and 3 prompting strategies. Our findings suggest that, on average, the non-native prompt performs the best, followed by mixed and native prompts.
BibTeX
@misc{kmainasi2024native,
  author = {Mohamed Bayan Kmainasi and Rakif Khan and Ali Ezzat Shahroor and Boushra Bendou and Maram Hasanain and Firoj Alam},
  title = {{Native vs Non-Native Language Prompting: A Comparative Analysis}},
  year = {2024},
  eprint = {2409.07054},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07054},
}
Figure