TL;DR
This paper introduces CEBiasBench, a new benchmark for evaluating cultural bias in LLMs, and MACD, a novel multi-agent debate framework that uses distinct cultural personas to achieve more culturally neutral outputs. They found that simply prompting LLMs in non-Western languages doesn't eliminate bias but shifts it, and MACD significantly outperforms baselines in mitigating this bias.
What Problem Does This Paper Address?
Large Language Models (LLMs) often exhibit a "Western-centric" bias, meaning their outputs tend to reflect Western values, norms, and perspectives, even when the prompt is culturally ambiguous or specifies a non-Western context. This is largely attributed to the data LLMs are trained on. While some research has explored mitigating this bias, existing methods have limitations:
- Evaluation: Current benchmarks often force responses into predefined cultural categories, lacking a "culturally neutral" option. This can mislabel unbiased responses as biased.
- Mitigation: Existing approaches either require massive, expensive multi-cultural datasets or use agent frameworks with functional roles (like "planner" or "critic") that don't explicitly represent diverse cultural viewpoints. This means they might not effectively address persistent cultural biases.
The authors also investigate a crucial question: does prompting LLMs in a non-Western language like Chinese actually eliminate Western bias, or does it simply shift the bias towards an East Asian perspective?
Methods
The paper proposes a two-pronged approach: a new evaluation framework and a novel mitigation method.
1. CEBiasBench and Multi-Agent Vote (MAV):
- CEBiasBench: A new bilingual (Chinese-English) benchmark designed to rigorously evaluate cultural bias. It covers five everyday cultural domains and includes questions that elicit culturally differentiated responses.
- Multi-Agent Vote (MAV): A stable evaluation protocol that uses multiple culturally diverse "judge" agents. These agents vote on whether a response exhibits cultural bias, with an explicit option for "no bias." This helps overcome the limitations of single-model evaluators and provides a more robust assessment.
2. Multi-Agent Cultural Debate (MACD):
- Training-Free Framework: MACD is designed to work at inference time without requiring additional model training.
- Distinct Cultural Personas: Instead of functional roles, MACD assigns agents specific cultural personas (e.g., Western, East Asian, African, Middle Eastern, South Asian). These personas are richly defined with background, values, and priorities.
- "Seeking Common Ground while Reserving Differences" (SCGRD): Agents engage in multi-round debates. In each round, they respond from their cultural standpoint, then observe and refine their responses based on others' perspectives. The SCGRD strategy encourages agents to find shared values and acknowledge compatible practices while preserving their unique cultural insights.
- Summary Model: After the debate, a summary model synthesizes the agents' converged viewpoints into a coherent, culturally balanced, and unbiased final response.
Datasets
- CEBiasBench: A newly introduced Chinese-English bilingual benchmark. It contains 100 questions per language across five cultural domains: Food Culture, Festivals and Celebrations, Etiquette and Social Norms, Daily Life Habits, and Traditions and Customs.
- CAMeL Benchmark: Used to test the cross-lingual generalization of MACD. Specifically, the culturally neutral subset CAMeL-Ag was utilized.
Key Insights and Ideas
- Language Switching Isn't Enough: Prompting LLMs in non-Western languages (like Chinese) doesn't eliminate cultural bias; it primarily shifts the bias towards the culture associated with that language (e.g., East Asian bias emerges). Western bias often persists.
- Explicit Cultural Personas Matter: Assigning agents distinct, well-defined cultural personas is crucial for surfacing and debating diverse viewpoints, leading to more balanced outputs. Functional roles alone are insufficient.
- Debate and Synthesis Drive Neutrality: The multi-round debate process, guided by the SCGRD strategy, allows agents to iteratively refine their responses, identify commonalities, and ultimately synthesize a culturally inclusive answer.
- Evaluator Bias is Real: Different LLM evaluators exhibit systematic biases in how they assign cultural labels. This highlights the importance of robust evaluation methods like MAV, which aggregate multiple judgments.
Key Results
- CEBiasBench Performance: On the GPT-4o backbone, MACD achieved a 57.6% average "No Bias" rate (LLM-as-judge) and an 86.0% "No Bias" rate (MAV), significantly outperforming direct generation (47.6% and 69.0%, respectively). Similar improvements were seen across other backbones and languages.
- Cross-Lingual Generalization: MACD demonstrated effectiveness on the Arabic CAMeL benchmark, achieving a 96.8% unbiased rate with the GPT-5 evaluator, outperforming direct generation by 14.4%.
- Bias Distribution: Analysis on CEBiasBench-CN showed that baseline models exhibited significant East Asian concentration (38%-48%) after Chinese prompting, while MACD reduced this bias and maintained low Western bias.
- Response Quality: MACD generated responses with higher information density and conciseness compared to baseline methods, indicating effective synthesis rather than information loss.
Limitations and Future Work
- Limited Scope of CEBiasBench: The current CEBiasBench covers five cultural domains and discrete labels, which may not capture the full spectrum of cultural nuances, intra-cultural diversity, or mixed responses. Expanding coverage to more regions, languages, and interaction types is a future direction.
- Repeated Evaluations: Due to API budget constraints, repeated multi-run evaluations were not conducted, and the stability of results needs further validation.
- Cultural Neutrality vs. Optimal Output: The authors acknowledge that cultural neutrality may not be universally optimal for all tasks or user preferences.
Future work could involve expanding the benchmark, exploring more complex cultural interactions, and investigating how MACD performs in more specialized or sensitive domains.
Summary updated