TL;DR
Large Language Models (LLMs) are increasingly expected to exhibit personality, but current evaluation methods often overlook cultural nuances, leading to Western-centric biases. This paper introduces CulturalPersonas, a novel benchmark designed to assess LLM personality expression across diverse cultural contexts, demonstrating that culturally grounded prompts significantly improve alignment with human behavior.
What Problem Does This Paper Address?
As LLMs become integrated into more interactive applications, their ability to express personality in a culturally appropriate manner is crucial. However, existing methods for evaluating LLM personality, such as those using the Big Five (OCEAN) framework, largely assume personality traits are universal. This assumption is challenged by decades of cultural psychology research, which shows that trait expression is deeply shaped by local norms, values, and communicative styles. Without accounting for these differences, LLM evaluations risk reinforcing Western behavioral norms and misrepresenting how personality is perceived globally. This paper aims to bridge this gap by providing a culturally sensitive benchmark for LLM personality evaluation.
Methods
The authors introduce CulturalPersonas, a large-scale benchmark comprising 3,000 scenario-based questions across six diverse countries: Brazil, India, Japan, Saudi Arabia, South Africa, and the United States. Each question targets one of the Big Five traits and is embedded in realistic, locally situated social contexts reflecting cultural values. The dataset was created using a retrieval-augmented generation (RAG) pipeline, seeded with academic cultural texts and validated by native annotators.
The evaluation framework uses two complementary formats:
- Multiple-Choice Selection (MCS): Models select from predefined Likert-style natural-language options. This provides a controlled environment for evaluating trait alignment.
- Open-Ended Generation (OEG): Models generate free-form behavioral responses, which are then mapped to trait scores. This format captures more naturalistic language and stylistic diversity.
To assess alignment, they use Wasserstein distance (W) and the Kolmogorov-Smirnov (KS) statistic, where lower values indicate better alignment with human personality distributions.
Datasets
- CulturalPersonas: A newly introduced benchmark consisting of 3,000 scenario-based questions across six countries, designed to evaluate LLM personality expression in culturally grounded contexts. It includes 500 questions per country, with 100 questions per OCEAN trait, and 60 real-world norms.
- IPIP-NEO dataset: Used as ground truth for human personality distributions, containing over 300,000 responses from global populations. Per-country subsets were extracted for the six countries in CulturalPersonas.
Key Insights and Ideas
- Cultural Nuance is Crucial: The study strongly demonstrates that personality expression is not universal. For example, assertiveness can be seen as high openness in Western cultures but disruptive in Japan, where harmony and indirectness are preferred.
- LLMs Struggle with Cultural Adaptation: Without explicit cultural priming, LLMs tend to default to more general or Western-centric responses, failing to capture culturally specific nuances.
- Scenario-Based Evaluation is Key: Moving beyond static questionnaires, behavior-rich scenarios grounded in local values are essential for a more accurate and meaningful evaluation of LLM personality.
- Cultural Priming Significantly Improves Alignment: Providing LLMs with country-specific persona prompts and cultural norms dramatically reduces the divergence from human personality distributions.
Key Results
- CulturalPersonas Improves Alignment: The benchmark consistently leads to better alignment with human personality distributions compared to existing psychometric tools (BFI, IPIP) and prior LLM benchmarks (TRAIT, Big5Chat), with reductions in Wasserstein distance of over 20% in some cases.
- OEG Reveals More Nuance: While MCS is easier for models, OEG highlights the challenges and successes in generating culturally appropriate free-form responses. CulturalPersonas elicits more expressive and lexically diverse outputs in OEG.
- Model Differences Emerge: The study reveals that different LLMs (GPT-4o-mini, Llama3, Qwen2) exhibit varying degrees of cultural adaptability, with some traits (e.g., Agreeableness, Extraversion) proving more challenging to model across cultures.
- Cultural Priming Boosts Performance: Removing cultural priming led to significant drops in alignment, particularly for traits like Agreeableness and Extraversion, underscoring the importance of contextual information.
Limitations and Future Work
- Cultures as Countries: The study acknowledges that using countries as a proxy for culture is an abstraction that simplifies diverse realities. Future work could explore finer-grained regional or individual differences.
- English-Only Dataset: The current benchmark is in English, which could be a confounding factor. Future work should expand to multilingual settings.
- Static Evaluation: The current evaluation captures personality at a single point in time. Future research could investigate temporal consistency and dynamic persona behavior.
- Imbalanced Country Datasets: The ground truth data for some countries (e.g., Saudi Arabia) is significantly smaller than others, which could impact the robustness of comparisons.
- Broader Representation: Future work should ensure broader representation of cultural norms beyond academic documents and explore other modalities for cultural expression.
- Responsible Use: The authors emphasize that findings should not be used to stereotype individuals and that personality is complex and influenced by many factors beyond nationality.
Summary updated