TL;DR
Most cultural evaluations of LLMs reduce a user's identity to a single label (e.g., "You are from China"). This paper explores "composite identities" by pitting a fixed system persona against conflicting user cues—like an Italian persona being told the user was born in Egypt or being addressed in Japanese. The authors find that country mentions and query languages are not interchangeable signals; while naming a country can nudge a model toward specific cultural values in open-ended text, changing the language often pushes the model's responses away from that culture’s typical distribution.
What Problem Does This Paper Address?
As of 2024, over 300 million people live outside their country of birth, often carrying multi-layered cultural identities. However, current research into "cultural alignment" for LLMs typically treats identity as a static, single variable.
The authors identify a critical gap: how do models handle conflicting cultural signals? For example, if a model's "memory" or system prompt identifies a user as Italian, but the user then says, "I was born in Egypt," or starts typing in Arabic, how does the model prioritize these cues? Understanding this is vital for building personalized assistants that don't fall into "one-size-fits-all" cultural stereotypes or exhibit inconsistent behavior across different languages.
Methods
The researchers stress-test three open-weight models (Llama-3.1-8B, Ministral-14B, and Gemma-3-27B) using a "composite-identity" setup.
- The Setup: They fix a "base identity" (persona) in the system prompt and introduce a second country identity in the user turn via two types of cues:
- Explicit Cues: Natural language statements like "I was born in [Country]."
- Implicit Cues: Changing the query language to the dominant language of [Country] without naming the country.
- The Tasks:
- Multiple-Choice Probing: Using 144 questions from the World Values Survey (WVS).
- Open-Ended Generation: Generating responses to sociopolitical topics derived from WVS.
- Value Extraction: For open-ended responses, they used an LLM-based annotator to extract "value expressions" (e.g., harmony, fairness, tradition) and mapped them to a taxonomy of 512 clusters.
- The Metric: They measured the "gap closed"—how much the cue moved the model's response from the base persona's distribution toward the target country's distribution.
Datasets
- World Values Survey (WVS): A standard social science instrument used to measure global opinions and values.
- IssueBench: A dataset of realistic prompts for measuring bias in LLM writing assistance, adapted here for sociopolitical topics.
- AIRiskDilemmas: A dataset of 2,999 dilemmas used to test if value shifts in text actually translate to different choices in AI behavior.
- WildChat/Disclosures: Used to source natural, first-person expressions of identity for the explicit cues.
Key Insights and Ideas
- Language is not a proxy for culture: One of the most striking "aha moments" was that posing a question in a country's language (e.g., Japanese) consistently moved multiple-choice answers away from that country's baseline distribution. This suggests that "language cues" might introduce cross-lingual noise or trigger different training data biases rather than activating "localized knowledge."
- Format dictates behavior: Models were much more responsive to explicit country cues in open-ended generation than in multiple-choice tasks. In free text, naming a country triggered highly specific cultural concepts—like whanaungatanga (kinship) for New Zealand—that were absent otherwise.
- Gemma vs. The Rest: The models didn't behave as a monolith. Gemma was the only model that consistently moved toward the target country in multiple-choice tasks when given an explicit cue, while Llama and Mistral often moved away.
Key Results
The study analyzed 1.6 million open-ended responses and nearly 8 million multiple-choice answers.
- Open-Ended Success: In free-text generation, explicit country cues closed 30–44% of the gap between the persona and the target country.
- The Language Divergence: In multiple-choice probing, language cues shifted distributions about three times more than explicit cues, but the shift was almost always in the "wrong" direction (away from the target).
- Signature Values: Naming specific countries significantly increased the frequency of certain values. For instance, naming Japan increased the expression of harmony by 21.9 percentage points.
Limitations and Future Work
- No "Correct" Answer: The authors emphasize that there is no ground-truth "correct" value distribution for a multicultural person. Their work characterizes how models change, not how they should change.
- Country $\neq$ Culture: Using "country" as a proxy for culture is a known simplification that ignores subcultures, generations, and socioeconomic diversity.
- Limited Scope: The study focused on only two simultaneous identities. Real-world intersectionality (religion, ethnicity, etc.) is far more complex.
- Future Direction: The authors suggest that future evaluations must treat language and country as independent variables rather than interchangeable proxies. Constructing evaluation data from genuinely bicultural or migrant individuals is a high-priority next step.
Summary updated