Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension

Amir Hossein Yari, Fajri Koto

Abstract

While multilingual Large Language Models (mLLMs) have demonstrated remarkable capabilities in general reasoning, their "procedural intelligence"—the ability to understand and execute sequences of actions—remains heavily biased toward Western contexts. This paper introduces CAPTex (Culturally-Aware Procedural Texts), a benchmark designed to evaluate mLLMs across 1,400 culturally specific procedures from seven diverse regions. Our analysis reveals significant performance disparities, particularly in low-resource languages and non-Western traditions, highlighting a critical "cultural blind spot" in current state-of-the-art models.

Introduction

Procedural texts, which outline sequences of actions to achieve a specific goal, are fundamental to human-computer interaction. However, these instructions are rarely culturally neutral. Whether performing a Shinto ritual in Japan or preparing a traditional Persian dish in Iran, the logic and sequence of steps are deeply rooted in local knowledge and traditions.

Current evaluations of Large Language Models (LLMs) often suffer from WEIRD bias (Western, Educated, Industrialized, Rich, and Democratic). When models lack an understanding of the cultural "why" or "how" behind a practice, they risk generating inaccurate, culturally insensitive, or nonsensical instructions. To address this, we present CAPTex, a multifaceted benchmark that quantifies the gap in procedural intelligence across diverse cultural domains.

The CAPTex Benchmark

CAPTex is a comprehensive dataset featuring 1,400 human-written procedures authored by native speakers from seven distinct regions: China, India, Indonesia, Iran, Japan, Nigeria, and Pakistan. The benchmark spans ten cultural domains, including food, festivals, social etiquette, craftsmanship, traditional attire, agriculture, religious practices, life milestones, sports, and environmental practices.

To ensure linguistic and cultural depth, each procedure exists in both its native language and a high-quality English translation. From these base procedures, the researchers derived 5,600 Multiple Choice Questions (MCQs) and 700 simulated conversations to test various dimensions of reasoning.

Methodology

The evaluation framework utilizes four distinct tasks to probe model understanding:

  1. Step Reordering: Models must reconstruct the correct sequence from a shuffled list of procedural steps.
  2. Procedure-Based MCQ (PB-MCQ): Models answer direct questions regarding the subsequent (what happens next) or antecedent (what happened before) steps in a procedure.
  3. Conversation-Based MCQ (CB-MCQ): Models must infer the next logical step within a simulated dialogue between two participants.
  4. Conversation-Based QA (CB-QA): A generative task where the model acts as a knowledgeable participant, providing contextually appropriate and natural responses in a dialogue.

Quality control was maintained through a two-step process involving automated structural consistency checks and manual peer reviews by native speakers to verify cultural authenticity.

Key Findings and Discussion

The Performance of State-of-the-Art Models

GPT-4o emerged as the gold standard, consistently outperforming other models across nearly all tasks and languages. It demonstrated the most balanced performance between English translations and native languages. Among open-weight models, Qwen2.5-14B-Instruct and Gemma-2-9b-it showed strong results, particularly in reordering and MCQ tasks.

The Language and Resource Gap

A significant decline in performance was observed in low-resource languages such as Hausa (Nigeria) and Urdu (Pakistan) compared to high-resource languages like Chinese or Japanese. This suggests that the "cultural blind spot" is exacerbated by a lack of linguistic representation in training data.

Directional Reasoning and Negation

Interestingly, models generally found it easier to reason about preceding steps (antecedents) than upcoming steps (subsequent). This indicates that LLMs are more adept at "looking back" to verify logic than "looking forward" to predict outcomes within a cultural context. Furthermore, while negation typically hinders model performance, CAPTex revealed that "Subsequent Negative" questions (asking what is not the next step) occasionally led to better performance, though "Affirmative Negative" questions remained a significant challenge.

The "Conversational Boost"

One of the most surprising insights was that models performed better on multiple-choice tasks when framed as a conversation (CB-MCQ) rather than direct questioning (PB-MCQ). Adopting the "persona" of a helpful respondent appears to help models leverage their reasoning capabilities more effectively than standard zero-shot prompting.

Domain-Specific Knowledge

Model proficiency is not uniform across a single culture. For example, Qwen2.5-14B-Instruct demonstrated deep understanding of Indian agriculture but struggled with Chinese agricultural topics, despite showing superior encoding of Chinese craftsmanship compared to other regions. This suggests that model knowledge is highly fragmented across different cultural domains.

Conclusion and Future Work

CAPTex highlights the necessity of moving beyond Western-centric evaluations to build truly global AI systems. While GPT-4o leads the field, the significant gaps in low-resource languages and specific cultural domains indicate that much work remains.

Future research should expand the geographic scope of such benchmarks and incorporate multimodal elements, such as images and video, to better reflect how humans learn and communicate procedures in the real world. Additionally, ensuring the safety and ethical accuracy of cultural instructions—particularly in sensitive areas like religious practices or healthcare—is a critical priority for the next generation of mLLMs.

Summary updated

BibTeX
@inproceedings{yari2025unveiling,
  author = {Amir Hossein Yari and Fajri Koto},
  title = {{Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure