The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities

Harsh Nishant Lalai, Raj Sanjay Shah, Jiaxin Pei, Sashank Varma, Yi-Chia Wang, Ali Emami

Authors: Harsh Nishant Lalai, Raj Sanjay Shah, Jiaxin Pei, Sashank Varma, Yi-Chia Wang, Ali Emami

Introduction

While Large Language Models (LLMs) are increasingly "aligned" to avoid explicit biases, these safety filters often act as a surface-level layer. Beneath this exterior, models frequently harbor implicit biases rooted in their training data. This research, presented at COLM 2025, investigates how geographic origin influences an LLM's ability to reason about and identify entities from different parts of the world.

Methodology: The 20 Questions Framework

To move beyond standard probing—which often triggers safety guardrails—the researchers employed a multi-turn deduction framework based on the game "20 Questions." In this setup, two instances of the same model interact: a "Judge" who knows the target entity and a "Guesser" who attempts to identify it through strategic questioning.

The 20 Questions deduction framework used to evaluate LLM reasoning.

The study evaluated three state-of-the-art models: GPT-4o-mini, Gemini 2.0 Flash, and Llama-3.3-70B-Instruct. These models were tested in both a "Canonical" (20-turn) and "Unlimited" (150-turn) setting across seven languages, including English, Hindi, Mandarin, and Spanish.

The Geo20Q+ Dataset

The researchers introduced Geo20Q+, a comprehensive benchmark containing 32,424 entities from 172 countries. The dataset is categorized into "Notable People" (e.g., politicians, athletes) and "Things" (e.g., culturally significant foods, landmarks, animals). Entities were sourced from Wikipedia and verified to ensure a clear primary country of origin.

Geographic Disparities in Success Rates

The results revealed a stark divide in performance based on geographic origin. Models were significantly more successful at deducing entities from the Global North and West compared to the Global South and East.

Global map illustrating the success and failure rates of LLMs in identifying entities based on their geographic origin.

For instance, in the canonical 20-turn setting, Gemini 2.0 Flash achieved a 30.9% success rate for notable people from the Global North, but only 10.2% for those from the Global South. A similar gap was observed between the Global West (23.4%) and the Global East (10.6%). Generally, models were better at identifying people (25-36% success) than culturally specific objects (9-13%).

Key Insights: The "Western Default"

The study uncovered a "Western Default" in the models' reasoning processes. When attempting to identify non-Western entities, models often systematically eliminated Western possibilities first. For example, when guessing an Indian poet, a model might first ask if the person is American, British, or French before considering Asian origins.

Furthermore, the researchers found that language is not a "silver bullet." Playing the game in a local language (e.g., Hindi for an Indian entity) did not significantly bridge the performance gap. In some cases, models were more accurate identifying Asian entities when playing in English than in the region's native language.

Conclusion

The findings suggest that geographic bias in LLMs is deeper than simple training data frequency. Even when models possess the necessary information (as evidenced in unlimited turn settings), they are less efficient at accessing it for the Global South. Future work may involve using interpretability tools to pinpoint the neural layers responsible for these geographic "blind spots" and exploring more balanced training datasets.

Summary updated

BibTeX
@inproceedings{lalai2025world,
  author = {Harsh Nishant Lalai and Raj Sanjay Shah and Jiaxin Pei and Sashank Varma and Yi-Chia Wang and Ali Emami},
  title = {{The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilities}},
  booktitle = {COLM},
  year = {2025},
  url = {https://openreview.net/forum?id=hJtvCfDfs1},
}
Figure