Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4

Zhuozhuo Joy Liu, Farhan Samir, Mehar Bhatia, Laura K. Nelson, Vered ShwartzarXiv 2025 · DOI · arXiv

Large Language Models (LLMs) like GPT-4 are increasingly used as global tools, yet their cultural foundations remain a subject of intense scrutiny. Recent research from the University of British Columbia and McGill University suggests that while these models are versatile, they are "Western by default." This study reveals that current safety guardrails may only be skin-deep, masking cultural stereotypes that persist within the model's underlying reasoning.

Methodology: From Surveys to Stories

Traditional evaluations of cultural bias often rely on a "top-down" approach, using direct questions from sociological instruments like the Hofstede Culture Survey. However, this method risks measuring a model's ability to parrot training data rather than its actual application of social norms. To address this, the researchers proposed a bottom-up narrative approach.

Figure 1: Comparison between traditional top-down survey methods and the proposed bottom-up narrative reasoning approach.

Instead of abstract questions, the study asked GPT-4 to derive social norms—expressed as "Rules of Thumb" (RoTs)—from movie plots across four distinct cultures: China, India, Iran, and the United States. By extracting norms from stories, the researchers could observe how the model applies cultural logic in context. They compared GPT-4's "default" responses to those generated under "cultural prompting" (e.g., "As someone living in China...") and a baseline of 88 human annotators from the target countries.

The "Western by Default" Phenomenon

A primary finding of the study is the "alignment gap." When GPT-4 is prompted without a specific cultural context, its judgments align most closely with US-based annotators. Using Jensen-Shannon Divergence (JSD) to measure the distance between the model's default stance and various cultural perspectives, the researchers found the model is closest to the US (0.12) and furthest from China (0.33), with Iran and India falling in between. This suggests that the "voice from nowhere" adopted by LLMs is, in fact, a Western voice.

Key Results: Accuracy vs. Specificity

The evaluation focused on three main metrics: accuracy, culture-specificity, and stereotypicality. While GPT-4 proved highly accurate in identifying relevant norms, it fell into what the authors call the "Generic Trap."

Figure 5: Quantitative comparison of Accuracy, Cultural-specificity, and Stereotypicality across GPT-4 variants and human baselines.

As shown in the results, GPT-4's accuracy is comparable to humans, but its "culture-specificity" is significantly lower. The model tends to generate universal platitudes (e.g., "It is good to help people") rather than the nuanced, specific norms that define a particular culture. Interestingly, while GPT-4's generated norms were less stereotypical than human-written ones—likely due to safety tuning—its internal biases remained detectable.

Stereotype Recovery and the Guardrail Problem

The most striking revelation of the study is "Stereotype Recovery." Although GPT-4's guardrails prevent it from volunteering overt stereotypes during generation, these biases surface when the model is asked to predict the opinions of non-Western groups.

For example, when asked to predict how a Chinese person would rate the statement "It is bad to work all the time," GPT-4 predicted they would strongly disagree, leaning into the "hardworking" stereotype. In reality, Chinese annotators were much more moderate in their views. This suggests that "preference tuning" may only be masking biases in the output layer while leaving stereotypical associations intact within the model's internal reasoning.

Conclusion

The study highlights the limitations of current LLM alignment. By moving to a bottom-up narrative evaluation, researchers have shown that GPT-4's cultural competence is often superficial and Western-centric. Future work must look beyond English-language Wikipedia and explore whether prompting in native languages can help models access more authentic cultural knowledge, ultimately moving toward AI that is truly globally representative rather than just "Western by default."

Summary updated

Abstract

LLMs have been demonstrated to align with the values of Western or North American cultures. Prior work predominantly showed this effect through leveraging surveys that directly ask (originally people and now also LLMs) about their values. However, it is hard to believe that LLMs would consistently apply those values in real-world scenarios. To address that, we take a bottom-up approach, asking LLMs to reason about cultural norms in narratives from different cultures. We find that GPT-4 tends to generate norms that, while not necessarily incorrect, are significantly less culture-specific. In addition, while it avoids overtly generating stereotypes, the stereotypical representations of certain cultures are merely hidden rather than suppressed in the model, and such stereotypes can be easily recovered. Addressing these challenges is a crucial step towards developing LLMs that fairly serve their diverse user base.
BibTeX
@misc{liu2025is,
  author = {Zhuozhuo Joy Liu and Farhan Samir and Mehar Bhatia and Laura K. Nelson and Vered Shwartz},
  title = {{Is It Bad to Work All the Time? Cross-Cultural Evaluation of Social Norm Biases in GPT-4}},
  year = {2025},
  eprint = {2505.18322},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.18322},
}
Figure