TL;DR
Large Language Models (LLMs) struggle with cultural nuances, particularly Persian "taarof" – a complex system of ritual politeness. This paper introduces TAAROFBENCH, the first benchmark for evaluating taarof understanding, and shows that current LLMs perform poorly, often mistaking politeness for cultural appropriateness. Fine-tuning significantly improves performance, highlighting the need for culturally aware LLMs.
What Problem Does This Paper Address?
LLMs are increasingly used in global contexts, but they often fail to grasp culturally specific communication norms. This can lead to misinterpretations, damaged relationships, and reinforced stereotypes. The paper focuses on "taarof," a sophisticated social norm in Iranian interactions characterized by deference, modesty, and indirectness. Taarof is absent from existing cultural benchmarks, and understanding it is crucial for effective cross-cultural communication. The authors aim to bridge this gap by creating a benchmark to evaluate LLMs' ability to understand and express taarof, and to explore methods for improving their cultural competence.
Methods
The core of this work is the TAAROFBENCH benchmark, a structured computational task that formalizes taarof interactions. It consists of 450 role-play scenarios covering 12 interaction topics, each annotated with culturally expected behavior derived from academic sources and validated by native Persian speakers. Scenarios are defined by environment, roles, context, user utterance, and the expected response pattern.
To evaluate LLMs, the authors used a zero-shot prompting format and GPT-4 as an external judge to assess response appropriateness against the expected cultural behavior. They also conducted controlled experiments to analyze the impact of language (English vs. Persian prompts), cultural context (explicit "in Iran" vs. no country reference), and gender on model performance.
For adaptation, they employed two main techniques:
- Supervised Fine-Tuning (SFT): Training the model on pairs of scenarios and culturally appropriate responses.
- Direct Preference Optimization (DPO): Training the model to prefer culturally appropriate responses over inappropriate ones.
Datasets
The primary dataset introduced is TAAROFBENCH, comprising 450 role-play scenarios designed to evaluate LLM understanding of Persian taarof. These scenarios were created from 30 distinct types of taarof identified in academic and cultural references. The benchmark was validated by native Persian speakers.
For adaptation experiments, a subset of TAAROFBENCH was used, with 345 scenarios for training and 105 for testing. This dataset was augmented with GPT-4 generated examples and manually filtered for quality.
A human study was conducted with 33 participants (native Persian, heritage Persian, and non-Iranian speakers) who responded to 30 scenarios to establish human performance baselines.
Key Insights and Ideas
- Politeness vs. Cultural Appropriateness: Standard politeness metrics are insufficient for evaluating cultural competence. LLMs can be rated as "polite" by general metrics while still violating specific cultural norms like taarof.
- Language as a Cultural Cue: Prompting LLMs in Persian significantly improves their understanding of taarof, suggesting that language itself carries strong cultural context.
- Smaller Models Rely on Explicit Cues: While larger models are less affected by the absence of explicit country references, smaller models show a notable decline in accuracy, indicating they rely more heavily on explicit cultural framing.
- Cultural Distance Matters: Human understanding of taarof decreases significantly with cultural distance, with native speakers performing much better than heritage and non-Iranian speakers. This highlights the depth of cultural knowledge required.
- Gendered Reasoning: Models sometimes exhibit gender bias, either by applying stereotypes or by assuming gender identities not specified in the prompt, even when taarof norms are gender-neutral.
Key Results
- LLM Performance on Taarof: All evaluated LLMs struggled significantly with taarof-expected scenarios, achieving accuracies between 34-42%, far below native human performance (81.8%). They performed much better on non-taarof scenarios (76-93%).
- Language Impact: Prompting in Persian led to substantial accuracy gains across all models, with DeepSeek V3 showing the largest improvement (+32.0 points).
- Human Baselines: Native Persian speakers achieved 81.8% accuracy on taarof-expected scenarios, setting a high ceiling. Heritage speakers scored 60.0%, and non-Iranians 42.3%.
- Topic-Specific Strengths: Models performed best on "gift" scenarios, likely due to their cross-cultural prevalence. "Making a request" and "compliment" scenarios posed the greatest challenge.
- Adaptation Gains: Fine-tuning significantly improved Llama 3's cultural alignment. SFT achieved a 20.0% overall accuracy gain, while DPO achieved a 33.3% gain, bringing performance on taarof-expected scenarios from 37.2% to 79.5%.
Limitations and Future Work
- Evolving Cultural Practices: TAAROFBENCH captures taarof as documented, but cultural practices evolve. Future work could explore how LLMs adapt to these shifts, potentially through continual learning.
- Broader Adaptation Potential: The current fine-tuning experiments show substantial gains with minimal data. Future work could explore multi-stage adaptation, culturally-specific pre-training, or methods that preserve cultural competence while learning new tasks.
- Cross-Cultural Transfer: The benchmark focuses on taarof. Future research could examine how learning one cultural norm affects performance on others, revealing if general cross-cultural competence can be developed or if each tradition requires dedicated adaptation.
- Interaction Complexity: The current methodology focuses on single-turn interactions. Extending to multi-turn exchanges would reveal if models can maintain cultural consistency over longer dialogues.
- Multimodal Cues: The text-based benchmark isolates verbal aspects. Future work could incorporate visual or auditory cues for a more holistic evaluation.
- Representation Risks: The authors acknowledge the risk of oversimplification and misrepresentation of cultural practices, which could reinforce stereotypes.
- Ethical Considerations: The paper raises concerns about privacy, data governance, responsible deployment, and the dual-use potential of cultural adaptation technologies.
Summary updated