CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks

Peiqin Lin, Chenyang Lyu, Wenjiang Luo, Haotian Ye, Md Mehrab Hossain, Chunlan Ma, Shaoxiong Ji, Younes Samih, Bo Zeng, Fan Jiang, Yuanbin Cao, Dilda Duisenbek, Adrian Neo Sau Xun, Daria Pozdniakova, Liubou Misevich, Nevena Marinković, Ngoc Gia Linh Nguyen, Thi Khanh Linh Do, Sarakmatak Sophy, Baotian Hu, Guanhua Chen, Gongbo Tang, Alham Fikri Aji, Longyue Wang, Weihua LuoarXiv 2026 · arXiv

TL;DR

While Large Language Models (LLMs) are increasingly global, they often fail to navigate the "grounded" cultural nuances of everyday life. CulturALL is a new benchmark of 2,610 tasks across 14 languages and 51 regions designed to test if models can reason through context-rich, culturally specific scenarios rather than just reciting trivia. The results are a wake-up call: even the most advanced models achieve less than 45% accuracy, revealing a significant gap in their ability to function as truly global assistants.

What Problem Does This Paper Address?

Most existing multilingual benchmarks test "encyclopedic trivia"—facts you could find in a textbook. For example, a model might know when the Qixi Festival is, but fail to help a florist in Beijing decide which flowers to stock for a late-August promotion based on that festival's traditions.

The authors argue that a truly capable LLM must master three intersecting axes:

  1. Multilingualism: Parsing the user's native tongue accurately.
  2. Multiculturalism: Accessing long-tail, domain-specific cultural facts.
  3. Grounded Reasoning: Synthesizing that knowledge to solve a real-world problem.

As shown in Figure 1, existing benchmarks like BLEND or NORMAD usually cover only one or two of these axes. CulturALL is the first to demand all three simultaneously.

Comparison of CulturALL with existing benchmarks across three axes.

Methods

The researchers developed a four-stage Human-LLM collaborative framework to build a dataset that is both diverse and genuinely difficult.

  1. Topic Sourcing: They identified 16 key topics (e.g., Commerce, Belief, Heritage, Finance) that impact daily life.
  2. Sample Creation: Annotators drew from personal experience, online resources (like Xiaohongshu), and existing datasets to draft scenarios.
  3. Sample Enrichment: This is the "secret sauce." To ensure the benchmark wasn't too easy, they used a "Difficulty Measure" where three LLMs attempted the questions. If the models got them right, human annotators "up-leveled" the samples using techniques like Long-Tail Swaps (replacing common entities with rarer ones) or Compositional Reasoning (merging two cultural knowledge points into one query).
  4. Release-Ready: A final stage of metadata completion and rigorous peer-review quality control.
The four-stage data construction framework of CulturALL.

Datasets

CulturALL consists of 2,610 samples spanning 14 languages and 51 regions. It covers 16 distinct topics to capture the full breadth of human activity.

Overview of CulturALL's coverage across languages, regions, and topics.

The authors also repurposed and "grounded" items from six existing public benchmarks, including GEOMLAMA, SeaEval, and Global-MMLU, by anchoring their knowledge requirements into concrete, narrative scenarios.

Key Insights and Ideas

  • The "Groundedness" Gap: Models that are "book smart" (good at trivia) are not necessarily "street smart" (good at grounded tasks). The reasoning required to apply a cultural fact to a specific scenario is a distinct and currently lacking capability.
  • Native Language Matters: Interestingly, testing models using English translations of the tasks resulted in an 8.08% drop in accuracy compared to using the original native-language prompts. This suggests that the native language itself carries essential cultural context that is lost in translation.
  • Reasoning vs. Retrieval: For some models (like Gemini), increasing reasoning effort and providing web search significantly boosted scores. However, for others (like GPT-5 or Claude-Opus), more reasoning tokens didn't lead to substantial gains, suggesting their underlying multi-step cultural reasoning is still fragile.
  • Open-Source Lag: There is a massive performance chasm between proprietary models and open-source alternatives. The best open-source model (Qwen series) trailed the top proprietary model by over 20% on "Easy" tasks and struggled significantly more on "Hard" ones.

Key Results

The benchmark proved to be exceptionally challenging:

  • Top Performer: Gemini-2.5-pro (with web search and "auto" reasoning) achieved the highest accuracy at 44.48%.
  • Open-Source Performance: The top-performing open-source configuration (Qwen3-235b) reached only 23.68%.
  • Difficulty Distribution: Over 55% of the samples were classified as "Hard" (solved by 4 or fewer of the 15 experimental settings). Even the best model only managed 18.47% accuracy on this Hard subset.
  • Language Variance: Performance was highly uneven across languages. Models generally performed better in Serbian and Bengali but struggled significantly in Japanese and Arabic.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Coverage Bias: While broad, some regions and topics remain underrepresented.
  • Regional vs. Intersectional Culture: The study defines culture primarily by geography, potentially missing nuances related to religion, age, or socio-economic status that cross borders.
  • Objective vs. Subjective: The benchmark relies on objective, verifiable answers for automatic evaluation. Future work should explore free-text generation to capture the nuances of cultural relativism.
  • Multimodality: CulturALL is text-only. Real-world cultural interaction involves visual symbols, gestures, and audio cues, which are currently out of scope.

Summary updated

Abstract

Large language models (LLMs) are now deployed worldwide, inspiring a surge of benchmarks that measure their multilingual and multicultural abilities. However, these benchmarks prioritize generic language understanding or superficial cultural trivia, leaving the evaluation of grounded tasks -- where models must reason within real-world, context-rich scenarios -- largely unaddressed. To fill this gap, we present CulturALL, a comprehensive and challenging benchmark to assess LLMs' multilingual and multicultural competence on grounded tasks. CulturALL is built via a human--AI collaborative framework: expert annotators ensure appropriate difficulty and factual accuracy, while LLMs lighten the manual workload. By incorporating diverse sources, CulturALL ensures comprehensive scenario coverage. Each item is carefully designed to present a high level of difficulty, making CulturALL challenging. CulturALL contains 2,610 samples in 14 languages from 51 regions, distributed across 16 topics to capture the full breadth of grounded tasks. Experiments show that the best LLM achieves 44.48% accuracy on CulturALL, underscoring substantial room for improvement.
BibTeX
@misc{lin2026culturall,
  author = {Peiqin Lin and Chenyang Lyu and Wenjiang Luo and Haotian Ye and Md Mehrab Hossain and Chunlan Ma and Shaoxiong Ji and Younes Samih and Bo Zeng and Fan Jiang and Yuanbin Cao and Dilda Duisenbek and Adrian Neo Sau Xun and Daria Pozdniakova and Liubou Misevich and Nevena Marinković and Ngoc Gia Linh Nguyen and Thi Khanh Linh Do and Sarakmatak Sophy and Baotian Hu and Guanhua Chen and Gongbo Tang and Alham Fikri Aji and Longyue Wang and Weihua Luo},
  title = {{CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Tasks}},
  year = {2026},
  eprint = {2604.19262},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.19262},
}

Loading related content...

Figure