Commonsense Reasoning in Arab Culture

Abdelrahman Boda Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, Fajri Koto

Abstract

Current Large Language Models (LLMs) often exhibit an "Anglocentric bias," failing to grasp the nuanced social norms and traditions of the Arab world. This paper introduces ArabCulture, a large-scale commonsense reasoning benchmark comprising 3,482 questions across 13 Arab countries. Unlike previous datasets derived from translation, ArabCulture was built from scratch by native speakers. Evaluation of 31 models reveals that while state-of-the-art models like GPT-4o perform well, significant gaps remain in regional cultural understanding, particularly in the Levant region.

Introduction: Addressing the "Cultural Gap"

Most evaluations of Arabic LLMs rely on machine-translated versions of English datasets, leading to models that understand Western holidays but struggle with the social etiquette of a Jordanian wedding or Algerian culinary traditions. Commonsense reasoning is deeply rooted in culture and geography. Without datasets reflecting local realities, we cannot accurately measure AI's utility for the 456 million people in the Arab world. The authors address this by moving away from translation and toward native-authored cultural grounding.

The ArabCulture Benchmark

The researchers developed ArabCulture, the first large-scale commonsense reasoning dataset built from scratch by native speakers. The dataset covers 13 countries across four regions: the Gulf, Levant, North Africa, and the Nile Valley.

Comparison of ArabCulture with existing Arabic benchmarks, highlighting its superior size and cultural focus.

Methodology

Moving away from standard web-scraping or translation, the authors employed a rigorous human-in-the-loop approach:

  1. Expert Authorship: 26 native speakers (two per country) authored original questions across 12 life domains (e.g., Food, Weddings, Habits).
  2. Quality Control: Questions underwent manual review and peer-validation to ensure clarity and cultural accuracy.
  3. Contextual Grounding: Models were tested with varying levels of location info (none, regional, or country-specific) to see if geographical context improves reasoning.
The ArabCulture dataset covers 13 countries with questions grounded in local traditions, as shown in these regional examples.

Evaluation and Results

The study evaluated 31 models, including multilingual giants (Llama-3, Qwen-2.5) and Arabic-specific models (Jais, AceGPT, ALLaM).

Key Findings

  • Top Performers: GPT-4o led the pack, achieving up to 90% accuracy with full context. Among open-weight models, Qwen-2.5-Instruct (72B) and AceGPT-v2-Chat (32B) were the strongest contenders, scoring around 80%.
  • The "English Prompt" Paradox: Surprisingly, many models performed better when prompted in English rather than Arabic, likely due to "instruction-tuning bias" where models are more heavily trained to follow directions in English.
  • The Levant Challenge: Across almost all models, the Levant region (Syria, Jordan, Palestine, Lebanon) proved the most difficult to reason about accurately, highlighting a specific gap in current training corpora.
  • Contextual Sensitivity: Providing specific country names improved performance for some models by over 6 percentage points, suggesting models possess latent knowledge but require geographical "nudges" to retrieve it.

Discussion and Conclusion

The results demonstrate that "common sense" is not universal. Scaling model size does not automatically improve cultural awareness; rather, the quality and diversity of pre-training data are paramount. While models like DeepSeek-R1 excel at logic, they struggle with cultural reasoning, suggesting these are distinct capabilities. Future work should focus on "culturally-aware instruction tuning" to help models better bridge the gap between their internal knowledge and regional social contexts.

Summary updated

BibTeX
@inproceedings{sadallah2025commonsense,
  author = {Abdelrahman Boda Sadallah and Junior Cedric Tonga and Khalid Almubarak and Saeed Almheiri and Farah Atif and Chatrine Qwaider and Karima Kadaoui and Sara Shatnawi and Yaser Alesh and Fajri Koto},
  title = {{Commonsense Reasoning in Arab Culture}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure