Large Language Models (LLMs) like GPT-4 are increasingly used as global tools, yet their cultural foundations remain a subject of intense scrutiny. Recent research from the University of British Columbia and McGill University suggests that while these models are versatile, they are "Western by default." This study reveals that current safety guardrails may only be skin-deep, masking cultural stereotypes that persist within the model's underlying reasoning.
Methodology: From Surveys to Stories
Traditional evaluations of cultural bias often rely on a "top-down" approach, using direct questions from sociological instruments like the Hofstede Culture Survey. However, this method risks measuring a model's ability to parrot training data rather than its actual application of social norms. To address this, the researchers proposed a bottom-up narrative approach.
Instead of abstract questions, the study asked GPT-4 to derive social norms—expressed as "Rules of Thumb" (RoTs)—from movie plots across four distinct cultures: China, India, Iran, and the United States. By extracting norms from stories, the researchers could observe how the model applies cultural logic in context. They compared GPT-4's "default" responses to those generated under "cultural prompting" (e.g., "As someone living in China...") and a baseline of 88 human annotators from the target countries.
The "Western by Default" Phenomenon
A primary finding of the study is the "alignment gap." When GPT-4 is prompted without a specific cultural context, its judgments align most closely with US-based annotators. Using Jensen-Shannon Divergence (JSD) to measure the distance between the model's default stance and various cultural perspectives, the researchers found the model is closest to the US (0.12) and furthest from China (0.33), with Iran and India falling in between. This suggests that the "voice from nowhere" adopted by LLMs is, in fact, a Western voice.
Key Results: Accuracy vs. Specificity
The evaluation focused on three main metrics: accuracy, culture-specificity, and stereotypicality. While GPT-4 proved highly accurate in identifying relevant norms, it fell into what the authors call the "Generic Trap."
As shown in the results, GPT-4's accuracy is comparable to humans, but its "culture-specificity" is significantly lower. The model tends to generate universal platitudes (e.g., "It is good to help people") rather than the nuanced, specific norms that define a particular culture. Interestingly, while GPT-4's generated norms were less stereotypical than human-written ones—likely due to safety tuning—its internal biases remained detectable.
Stereotype Recovery and the Guardrail Problem
The most striking revelation of the study is "Stereotype Recovery." Although GPT-4's guardrails prevent it from volunteering overt stereotypes during generation, these biases surface when the model is asked to predict the opinions of non-Western groups.
For example, when asked to predict how a Chinese person would rate the statement "It is bad to work all the time," GPT-4 predicted they would strongly disagree, leaning into the "hardworking" stereotype. In reality, Chinese annotators were much more moderate in their views. This suggests that "preference tuning" may only be masking biases in the output layer while leaving stereotypical associations intact within the model's internal reasoning.
Conclusion
The study highlights the limitations of current LLM alignment. By moving to a bottom-up narrative evaluation, researchers have shown that GPT-4's cultural competence is often superficial and Western-centric. Future work must look beyond English-language Wikipedia and explore whether prompting in native languages can help models access more authentic cultural knowledge, ultimately moving toward AI that is truly globally representative rather than just "Western by default."
Summary updated