Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation

Mehwish Nasim, Sanjeevan Selvaganapathy, Neel Ganapathi Sabhahit, Marie Griesbach, Pranav Bhandari, Janina Lütke Stockdiek, Lennart Schäpermeier, Usman Naseem, Christian GrimmearXiv 2026 · arXiv

TL;DR

While large language models (LLMs) excel at reciting factual cultural knowledge, they suffer from a massive "deployment gap" when it comes to changing how they speak based on situational context. This paper introduces a novel evaluation framework showing that LLMs recover only about one-fifth of their explicit cultural adaptation capabilities when relying on implicit situational cues alone.

What Problem Does This Paper Address?

Existing cultural benchmarks for LLMs typically evaluate factual knowledge (e.g., "What is a traditional wedding custom in India?") or explicit instruction-following (e.g., "Respond to this email using a highly deferential tone"). However, human communication relies heavily on cultural pragmatics—the unwritten social rules dictating how we adapt our language based on implicit situational context. For instance, a human instinctively knows to speak differently to a corporate executive than to a close family member, even without being explicitly ordered to do so.

The authors tackle a fundamental question: If an LLM can produce a culturally appropriate style when told explicitly, will it do so when the same context is only implied by the situation? Understanding this is crucial for deploying LLMs in sensitive real-world applications like healthcare, education, and customer support, where an answer can be factually correct yet socially jarring.

Methods

The authors design a rigorous three-prompt triad evaluation framework to isolate implicit pragmatic adaptation:

  1. Prompt A (Neutral Baseline): Presents a pragmatic dilemma with no extra cultural cueing to measure the model's default behavior.
  2. Prompt B (Explicit Instruction / Upper Bound): Appends a direct instruction detailing the target cultural orientation (e.g., "Respond as someone who strongly values hierarchical harmony").
  3. Prompt C (Implicit Cueing / The Test): Omits the explicit instruction but adds situational details that imply the same cultural stakes (e.g., mentioning that a boss personally developed a workflow, making public disagreement face-threatening).
Figure 1: Three-prompt design used throughout the paper. Prompt A is the neutral baseline, Prompt B adds an explicit cultural instruction, and Prompt C adds only implicit situational cueing. PCS asks how much of the Prompt A→B shift is recovered in Prompt A→C.

To quantify performance, they define Pragmatic Context Sensitivity (PCS) as the fraction of the explicit shift (Prompt A $\rightarrow$ B) that naturally reappears under implicit cueing (Prompt A $\rightarrow$ C):

$$\text{PCS} = \frac{\bar{C} - \bar{A}}{\bar{B} - \bar{A}}$$

The evaluation maps onto three cultural dimensions from the Hofstede tradition: Power Distance (PDI), Individualism-Collectivism (IDV), and Uncertainty Avoidance (UAI), scored across 12 distinct linguistic features using a validated three-LLM judge ensemble.

Datasets

The authors constructed a new multilingual benchmark consisting of:

  • 60 culturally grounded conversational scenarios spanning four domains: Workplace, Family, Social, and Institutional.
  • 5 evaluation languages: English, German, Hindi, Nepali, and Urdu.
  • 14,400 total generated responses (60 scenarios $\times$ 5 languages $\times$ 4 models $\times$ 3 prompts $\times$ 4 samples).

Crucially, the dataset leverages Hindi and Urdu as a natural control group. These languages share core grammar and semantics but map onto distinct cultural communities and use different scripts, allowing researchers to isolate whether models respond to linguistic structure or cultural indexicality.

Key Insights and Ideas

  • The Implicit-Explicit Gap is Severe: Across all models and languages, LLMs struggle to translate implicit situational awareness into pragmatic behavioral changes. They know the cultural rules, but they don't apply them unless explicitly commanded.
  • Hierarchy Cues are Easier to Read: Transfer is significantly stronger for authority-related cues (Power Distance) than for group-oriented ones (Individualism-Collectivism). The authors hypothesize that hierarchy provides highly salient, redundant surface signals (e.g., public vs. private settings, clear seniority roles) that are easier for models to parse.
  • Alignment Training Suppresses Uncertainty: In all five languages, explicitly prompting models to show high uncertainty avoidance (hedging) actually caused them to hedge less. This reveals a direct conflict between cultural pragmatics and RLHF (Reinforcement Learning from Human Feedback), which heavily optimizes for crisp, confident, and helpful assistant personas.
  • Grammar Trumps Culture: The Hindi-Urdu natural experiment revealed no reliable baseline difference in pragmatic behavior. This suggests that LLMs default to patterns dictated by shared linguistic and grammatical structures rather than the distinct cultural associations a specific script or register carries.
Figure 2: One illustrative cell from the released results: Hindi, IDV, outcome framing. The explicit shift (Prompt A→B) is large, while the implicit shift (Prompt A→C) is small, yielding stable PCS = 0.06.

Key Results

  • Overall Performance: The primary stable-only PCS mean across all models was 0.196, meaning models only recover roughly 20% of their explicit cultural adaptation capability when relying on implicit context.
  • Dimension Asymmetry: Implicit transfer was highest for Power Distance (0.299), intermediate for Uncertainty Avoidance (0.161), and lowest for Individualism-Collectivism (0.120).
Figure 3: Mean stable-only PCS by language and prag- matic dimension. Power Distance is highest across all five languages. UAI averages exclude the unsta- ble hedging-density feature.
  • Feature Breakdown: Authority-related features like deference (0.377) and face-saving (0.370) dominated the top of the implicit sensitivity rankings, while agency attribution (0.068) was the weakest.
Figure 4: Feature-level implicit sensitivity using median stable PCS across languages. Authority-related features dominate the top of the ranking. Hedging density is shown as unstable rather than assigned a primary PCS value.
  • Model Variance: Frontier models performed slightly better but still fell short. Gemini-3-flash achieved the highest overall PCS (0.244), followed by Mimo-V2-flash (0.222), Grok-4.1-fast (0.172), and Ministral-8B (0.157).

Limitations and Future Work

The authors note several limitations in their current approach:

  • Controlled Probes vs. Natural Conversation: The 60 scenarios are synthetic, single-turn prompts designed to isolate variables, meaning they lack the rich discourse history and turn-by-turn dynamics of real-world interactions.
  • Limited Language and Model Scope: While multilingual, the language set is genealogically narrow, and the evaluation is restricted to four closed-source models.
  • Hofstede Framework Constraints: Relying on national-level cultural dimensions risks essentializing culture and ignoring individual variation.

Future Directions: The authors suggest that future work should validate these findings using naturalistic conversational data and explore how assistant-tuning and RLHF paradigms can be modified to allow implicit context to override rigid, homogenized default personas.

Summary updated

Abstract

Many benchmarks show that large language models can answer direct questions about culture. We study a different question: do they also change how they speak when culture is only implied by the situation? We evaluate 60 culturally grounded conversational scenarios across five languages in three conditions: a neutral baseline (Prompt A), an explicit cultural instruction (Prompt B), and implicit situational cueing (Prompt C). We score responses on 12 pragmatic features covering deference to authority, individual-versus-group framing, and uncertainty management. We define Pragmatic Context Sensitivity (PCS) as the fraction of the Prompt A->B shift that reappears under Prompt A->C. Across four deployed LLMs and five languages (English, German, Hindi, Nepali, Urdu), the primary stable-only PCS mean is 0.196 (SD = 0.113), indicating that the models recover only about one-fifth of the pragmatic shift they can produce when instructed explicitly. Transfer is strongest for authority-related cues (0.299) and weakest for individual-versus-group framing (0.120). Uncertainty-related behaviour is mixed: hedging density exhibits negative explicit gaps in all five languages, suggesting that alignment training actively suppresses the target behaviour. Because Hindi and Urdu share core grammar yet index distinct cultural communities, we use them as a natural control; a paired analysis finds no reliable baseline difference (t = 0.96, p = 0.339, dz = 0.06), suggesting that models respond primarily to linguistic structure rather than to the cultural associations a language carries. We argue that multilingual cultural pragmatics is an explicit-versus-implicit deployment problem, not only a factual knowledge problem.
BibTeX
@misc{nasim2026do,
  author = {Mehwish Nasim and Sanjeevan Selvaganapathy and Neel Ganapathi Sabhahit and Marie Griesbach and Pranav Bhandari and Janina Lütke Stockdiek and Lennart Schäpermeier and Usman Naseem and Christian Grimme},
  title = {{Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation}},
  year = {2026},
  eprint = {2604.17718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.17718},
}

Loading related content...

Figure