TL;DR
While Large Language Models (LLMs) are increasingly global, they often fail to navigate the "grounded" cultural nuances of everyday life. CulturALL is a new benchmark of 2,610 tasks across 14 languages and 51 regions designed to test if models can reason through context-rich, culturally specific scenarios rather than just reciting trivia. The results are a wake-up call: even the most advanced models achieve less than 45% accuracy, revealing a significant gap in their ability to function as truly global assistants.
What Problem Does This Paper Address?
Most existing multilingual benchmarks test "encyclopedic trivia"—facts you could find in a textbook. For example, a model might know when the Qixi Festival is, but fail to help a florist in Beijing decide which flowers to stock for a late-August promotion based on that festival's traditions.
The authors argue that a truly capable LLM must master three intersecting axes:
- Multilingualism: Parsing the user's native tongue accurately.
- Multiculturalism: Accessing long-tail, domain-specific cultural facts.
- Grounded Reasoning: Synthesizing that knowledge to solve a real-world problem.
As shown in Figure 1, existing benchmarks like BLEND or NORMAD usually cover only one or two of these axes. CulturALL is the first to demand all three simultaneously.
Methods
The researchers developed a four-stage Human-LLM collaborative framework to build a dataset that is both diverse and genuinely difficult.
- Topic Sourcing: They identified 16 key topics (e.g., Commerce, Belief, Heritage, Finance) that impact daily life.
- Sample Creation: Annotators drew from personal experience, online resources (like Xiaohongshu), and existing datasets to draft scenarios.
- Sample Enrichment: This is the "secret sauce." To ensure the benchmark wasn't too easy, they used a "Difficulty Measure" where three LLMs attempted the questions. If the models got them right, human annotators "up-leveled" the samples using techniques like Long-Tail Swaps (replacing common entities with rarer ones) or Compositional Reasoning (merging two cultural knowledge points into one query).
- Release-Ready: A final stage of metadata completion and rigorous peer-review quality control.
Datasets
CulturALL consists of 2,610 samples spanning 14 languages and 51 regions. It covers 16 distinct topics to capture the full breadth of human activity.
The authors also repurposed and "grounded" items from six existing public benchmarks, including GEOMLAMA, SeaEval, and Global-MMLU, by anchoring their knowledge requirements into concrete, narrative scenarios.
Key Insights and Ideas
- The "Groundedness" Gap: Models that are "book smart" (good at trivia) are not necessarily "street smart" (good at grounded tasks). The reasoning required to apply a cultural fact to a specific scenario is a distinct and currently lacking capability.
- Native Language Matters: Interestingly, testing models using English translations of the tasks resulted in an 8.08% drop in accuracy compared to using the original native-language prompts. This suggests that the native language itself carries essential cultural context that is lost in translation.
- Reasoning vs. Retrieval: For some models (like Gemini), increasing reasoning effort and providing web search significantly boosted scores. However, for others (like GPT-5 or Claude-Opus), more reasoning tokens didn't lead to substantial gains, suggesting their underlying multi-step cultural reasoning is still fragile.
- Open-Source Lag: There is a massive performance chasm between proprietary models and open-source alternatives. The best open-source model (Qwen series) trailed the top proprietary model by over 20% on "Easy" tasks and struggled significantly more on "Hard" ones.
Key Results
The benchmark proved to be exceptionally challenging:
- Top Performer: Gemini-2.5-pro (with web search and "auto" reasoning) achieved the highest accuracy at 44.48%.
- Open-Source Performance: The top-performing open-source configuration (Qwen3-235b) reached only 23.68%.
- Difficulty Distribution: Over 55% of the samples were classified as "Hard" (solved by 4 or fewer of the 15 experimental settings). Even the best model only managed 18.47% accuracy on this Hard subset.
- Language Variance: Performance was highly uneven across languages. Models generally performed better in Serbian and Bengali but struggled significantly in Japanese and Arabic.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Coverage Bias: While broad, some regions and topics remain underrepresented.
- Regional vs. Intersectional Culture: The study defines culture primarily by geography, potentially missing nuances related to religion, age, or socio-economic status that cross borders.
- Objective vs. Subjective: The benchmark relies on objective, verifiable answers for automatic evaluation. Future work should explore free-text generation to capture the nuances of cultural relativism.
- Multimodality: CulturALL is text-only. Real-world cultural interaction involves visual symbols, gestures, and audio cues, which are currently out of scope.
Summary updated