Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs

Mahammed Kamruzzaman, Gene Louis KimEMNLP 2025

TL;DR

This paper investigates how assigning nationality personas to Large Language Models (LLMs) influences their perceptions of different countries. The study reveals that while LLMs generally exhibit a pro-Western bias, nationality personas can shift focus towards the persona's own region, sometimes amplifying existing stereotypes. This work highlights the need for careful consideration of persona design to ensure fairness and avoid overgeneralization in LLM outputs.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being customized for specific tasks and contexts through persona assignment. While this personalization can enhance user experience, it raises critical questions about how these personas, particularly nationality-based ones, might influence the LLM's "perception" or evaluation of different nations. The authors are concerned that these nationality personas could either serve as a mitigation for existing biases in LLMs or, conversely, amplify them, leading to unfair representations and perpetuating harmful stereotypes across international lines. Understanding this dynamic is crucial as LLMs are increasingly used for information synthesis at a global level. The paper aims to explore whether LLMs, when adopting a national persona, treat all regions and groups fairly or exhibit systematic favoritism.

Methods

The core of this research involves assigning 193 different nationality personas to five LLMs (GPT-4o, Llama-3.1-70B, Mistral-7B, Gemma2-27B, and Qwen2-7B). For each persona-LLM combination, the models were prompted with nation-describing adjectives (both positive and negative) to generate country names that came to mind.

Two key metrics were used to analyze the LLM outputs:

  • Response Percentage (RP): This measures the frequency with which LLMs generated responses associated with specific regions or countries, indicating when nations are represented.
  • Positively Mentioned Rate (PMR): This quantifies the proportion of positive adjective prompts that resulted in a country being mentioned, indicating how nations are represented (positively or negatively).

To ensure comparability, the dataset was down-sampled to balance adjective lists and region sizes. The study also compared LLM-generated perceptions with human survey data to assess alignment with real-world views.

Datasets

The study utilized a dataset of nation-describing adjectives collected from describingwords.io. This initial list of 1,000 adjectives was filtered down to 516 (316 positive, 200 negative) through a consensus-based process involving four human annotators. The filtering rules excluded adjectives that directly referenced a nation, were ambiguous in sentiment, or were otherwise unsuitable.

Human perception data was sourced from two surveys: the Pew Research Center and Gallup.

Key Insights and Ideas

  • The "Western Gaze" Persists: Even with nationality personas, LLMs consistently show a bias towards Western European nations, suggesting a deeply ingrained tendency in their underlying training data.
  • Persona-Driven Regional Focus: Nationality personas significantly influence LLMs to focus more on and favor countries within the persona's own region. For instance, an American persona is more likely to mention other North American countries.
  • Amplification of Stereotypes: While personas can shift focus, they don't necessarily correct biases. Instead, they can amplify existing stereotypes, leading to more negative portrayals of regions like Eastern Europe, Latin America, and Africa.
  • Nuance in Human Alignment: LLMs' generated perceptions correlate with human survey data, but the absolute values don't always match. This suggests LLMs partially reproduce human biases rather than perfectly mirroring them.
  • Model Variability: Different LLMs exhibit varying degrees of sensitivity to nationality personas and display distinct patterns in their biases and refusal rates. For example, Qwen2 showed a higher RP for Latin American and African states and a tendency not to refuse negative prompts.

Key Results

  • Pro-Western Bias: Across all models and personas, Western European nations were the most frequently mentioned (around 46% of responses), while Latin American and African states were least responded to.
  • Persona Influence: When a persona from a specific region was assigned, responses from that region increased significantly (e.g., Eastern European personas led to ~45% responses from Eastern European regions). The increase in RP for a persona's own country was about 17.3%, and for its region, it was 34.5%.
  • Negative Portrayals: Eastern European countries consistently received the lowest PMR values, indicating negative treatment. Latin American and African states, despite lower RP, had higher PMR values than Eastern European states, suggesting a complex interplay of representation and sentiment.
  • Human Alignment: Spearman correlations between LLM outputs and human survey data ranged from 0.76 to 0.88 for US perceptions of other nations, indicating a reasonable alignment. However, for other nations' perceptions of the US, the correlation was weaker or non-existent for some models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Experimental Design: The study used a simplified, non-ecologically valid setting to isolate biases. Real-world LLM behavior might differ due to the interaction of biases with other contextual variables.
  • Single-Trait Personas: The use of single-trait nationality personas oversimplifies the diversity within nationalities. Future work should explore multi-trait personas (e.g., combining nationality with gender or age) to capture more complex interactions.
  • English-Only Experiments: The reliance on English prompts limits the scope, as language can be intertwined with national and cultural identities.
  • Limited Number of LLMs: Testing only five models provides a snapshot; broader testing could offer a more complete picture.
  • No Mitigation Strategies: The study identifies biases but does not propose specific solutions. Future work should focus on developing targeted interventions to reduce biases and improve LLM alignment with ethical standards.
  • Training Data Representation: While the overrepresentation of Western European nations might stem from training data, the study suggests persona influence interacts with these biases in complex ways, warranting further investigation into dataset composition and fine-tuning.

Future work could also explore finer-grained sub-regional analyses and develop methods to quantify and mitigate underrepresentation at both data and model levels.

Summary updated

BibTeX
@inproceedings{kamruzzaman2025exploring,
  author = {Mahammed Kamruzzaman and Gene Louis Kim},
  title = {{Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure