Abstract
Current Large Language Models (LLMs) often exhibit an "Anglocentric bias," failing to grasp the nuanced social norms and traditions of the Arab world. This paper introduces ArabCulture, a large-scale commonsense reasoning benchmark comprising 3,482 questions across 13 Arab countries. Unlike previous datasets derived from translation, ArabCulture was built from scratch by native speakers. Evaluation of 31 models reveals that while state-of-the-art models like GPT-4o perform well, significant gaps remain in regional cultural understanding, particularly in the Levant region.
Introduction: Addressing the "Cultural Gap"
Most evaluations of Arabic LLMs rely on machine-translated versions of English datasets, leading to models that understand Western holidays but struggle with the social etiquette of a Jordanian wedding or Algerian culinary traditions. Commonsense reasoning is deeply rooted in culture and geography. Without datasets reflecting local realities, we cannot accurately measure AI's utility for the 456 million people in the Arab world. The authors address this by moving away from translation and toward native-authored cultural grounding.
The ArabCulture Benchmark
The researchers developed ArabCulture, the first large-scale commonsense reasoning dataset built from scratch by native speakers. The dataset covers 13 countries across four regions: the Gulf, Levant, North Africa, and the Nile Valley.
Methodology
Moving away from standard web-scraping or translation, the authors employed a rigorous human-in-the-loop approach:
- Expert Authorship: 26 native speakers (two per country) authored original questions across 12 life domains (e.g., Food, Weddings, Habits).
- Quality Control: Questions underwent manual review and peer-validation to ensure clarity and cultural accuracy.
- Contextual Grounding: Models were tested with varying levels of location info (none, regional, or country-specific) to see if geographical context improves reasoning.
Evaluation and Results
The study evaluated 31 models, including multilingual giants (Llama-3, Qwen-2.5) and Arabic-specific models (Jais, AceGPT, ALLaM).
Key Findings
- Top Performers: GPT-4o led the pack, achieving up to 90% accuracy with full context. Among open-weight models, Qwen-2.5-Instruct (72B) and AceGPT-v2-Chat (32B) were the strongest contenders, scoring around 80%.
- The "English Prompt" Paradox: Surprisingly, many models performed better when prompted in English rather than Arabic, likely due to "instruction-tuning bias" where models are more heavily trained to follow directions in English.
- The Levant Challenge: Across almost all models, the Levant region (Syria, Jordan, Palestine, Lebanon) proved the most difficult to reason about accurately, highlighting a specific gap in current training corpora.
- Contextual Sensitivity: Providing specific country names improved performance for some models by over 6 percentage points, suggesting models possess latent knowledge but require geographical "nudges" to retrieve it.
Discussion and Conclusion
The results demonstrate that "common sense" is not universal. Scaling model size does not automatically improve cultural awareness; rather, the quality and diversity of pre-training data are paramount. While models like DeepSeek-R1 excel at logic, they struggle with cultural reasoning, suggesting these are distinct capabilities. Future work should focus on "culturally-aware instruction tuning" to help models better bridge the gap between their internal knowledge and regional social contexts.
Summary updated