LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations

Viet-Thanh Pham, Lizhen Qu, Thuy-Trang Vu, Gholamreza Haffari, Dinh Phung

TL;DR

LiveCultureBench is a first-of-its-kind dynamic benchmark that evaluates LLM agents not just on their ability to complete tasks, but on their adherence to socio-cultural norms in a simulated town. By embedding models in a multi-agent environment with realistic social pressures, the researchers found that even top-tier models often prioritize efficiency over cultural appropriateness, especially in diverse, multicultural settings.

What Problem Does This Paper Address?

As we move toward a world of autonomous AI agents—think digital assistants that can book your travel or AI NPCs in video games—we need them to do more than just "succeed" at a task. They need to behave in ways that are socially and culturally acceptable.

Current benchmarks for Large Language Models (LLMs) have two major flaws:

  1. They are static: Most tests use simple Q&A (e.g., "Is it rude to do X in country Y?"). This doesn't capture how an agent might behave over a long day of interactions where goals and social norms conflict.
  2. They are culturally "flat": Most simulations assume a Western-centric or culturally homogeneous world.

The authors argue that we need to see how agents handle "norm-sensitive" behavior—balancing the need to get things done with the need to follow the unwritten rules of the society they inhabit.

Methods

The researchers built a "small city" modeled as a graph of locations (homes, offices, parks, etc.).

  • The Simulation: They populated this town with synthetic residents. One "Target Agent" (the LLM being tested) is given a daily goal and subtasks. The other "Supporting Agents" provide social context and, crucially, are programmed to apply "social pressure"—nudging or tempting the target agent to break local norms.
  • The Evaluator: Instead of just checking if a task was finished, a separate Verifier Agent (another LLM) monitors every action. It scores the target on goal completion, norm violations, faithfulness to their persona, and contextual awareness.
  • Uncertainty Management: Recognizing that "LLM-as-a-judge" can be unreliable, the authors used Conformal Prediction. This technique allows the system to estimate the "risk" of an incorrect judgment, providing a more trustworthy automated evaluation.

Datasets

  • CultureBank: The primary source for high-quality, human-annotated cultural norms.
  • Australian Census (Melbourne): To make the simulation realistic, the authors didn't just invent people. they sampled demographic distributions (age, gender, occupation, nationality) from real-world data for Melbourne, a city known for its multiculturalism.
  • LiveCultureBench: The paper introduces this entire framework as a new benchmark for the community.

Key Insights and Ideas

  • The Efficiency Trap: LLM agents are often "willing to trade cultural appropriateness for task completion." When a norm makes a task harder or slower, models frequently choose to break the rule to "get the job done."
  • Home vs. Public Life: Models are much better at following norms in "private" settings like apartments. The authors suggest this is because these settings often involve family interactions (usually a single culture), whereas public spaces like offices or malls involve more complex, diverse interactions where models stumble.
  • Pretraining Biases: Models within the same family (e.g., the Gemini series) show very similar cultural bias patterns. They tend to be more robust in cultures heavily represented in their training data (like British or German) and struggle significantly with others (like Vietnamese or Filipino).
  • Diversity Degrades Performance: As the number of different cultures in a social interaction increases, the target agent’s ability to adhere to norms drops consistently.

Key Results

  • Top Performer: Gemini-2.5-Pro emerged as the strongest overall model, balancing task success with norm adherence better than its peers.
  • The "Strong but Rude" Agent: Ministral 3-14B was highly effective at completing goals and maintaining context but performed poorly on norm adherence and profile faithfulness.
  • Verifier Accuracy: The automated Verifier Agent was remarkably effective, achieving F1 scores between 89% and 96% across various evaluation tasks when compared to human ground truth.
  • Multicultural Decline: In scenarios with 7+ cultures, norm adherence scores dropped by roughly 15-30% compared to single-culture environments for most models.

Limitations and Future Work

The authors are transparent about several hurdles:

  • Cultural Coverage: The benchmark is limited by the availability of structured cultural norm data. Many cultures and ethnicities are not yet represented.
  • The "Judge" Problem: While conformal prediction helps, an LLM judge still cannot fully replace human nuance in sensitive cultural evaluations.
  • Compute Costs: Running thousands of multi-agent simulations is computationally expensive, which might limit accessibility for smaller research teams.

Future work aims to incorporate community feedback to expand the norm database and develop even more robust mechanisms for representing uncertainty in social judgments.

Summary updated

Abstract

Large language models (LLMs) are increasingly deployed as autonomous agents, yet evaluations focus primarily on task success rather than cultural appropriateness or evaluator reliability. We introduce LiveCultureBench, a multi-cultural, dynamic benchmark that embeds LLMs as agents in a simulated town and evaluates them on both task completion and adherence to socio-cultural norms. The simulation models a small city as a location graph with synthetic residents having diverse demographic and cultural profiles. Each episode assigns one resident a daily goal while others provide social context. An LLM-based verifier generates structured judgments on norm violations and task progress, which we aggregate into metrics capturing task-norm trade-offs and verifier uncertainty. Using LiveCultureBench across models and cultural profiles, we study (i) cross-cultural robustness of LLM agents, (ii) how they balance effectiveness against norm sensitivity, and (iii) when LLM-as-a-judge evaluation is reliable for automated benchmarking versus when human oversight is needed.
BibTeX
@inproceedings{pham2026liveculturebench,
  author = {Viet-Thanh Pham and Lizhen Qu and Thuy-Trang Vu and Gholamreza Haffari and Dinh Phung},
  title = {{LiveCultureBench: a Multi-Agent, Multi-Cultural Benchmark for Large Language Models in Dynamic Social Simulations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1247},
}
Figure