Authors: Harsh Nishant Lalai, Raj Sanjay Shah, Jiaxin Pei, Sashank Varma, Yi-Chia Wang, Ali Emami
Introduction
While Large Language Models (LLMs) are increasingly "aligned" to avoid explicit biases, these safety filters often act as a surface-level layer. Beneath this exterior, models frequently harbor implicit biases rooted in their training data. This research, presented at COLM 2025, investigates how geographic origin influences an LLM's ability to reason about and identify entities from different parts of the world.
Methodology: The 20 Questions Framework
To move beyond standard probing—which often triggers safety guardrails—the researchers employed a multi-turn deduction framework based on the game "20 Questions." In this setup, two instances of the same model interact: a "Judge" who knows the target entity and a "Guesser" who attempts to identify it through strategic questioning.
The study evaluated three state-of-the-art models: GPT-4o-mini, Gemini 2.0 Flash, and Llama-3.3-70B-Instruct. These models were tested in both a "Canonical" (20-turn) and "Unlimited" (150-turn) setting across seven languages, including English, Hindi, Mandarin, and Spanish.
The Geo20Q+ Dataset
The researchers introduced Geo20Q+, a comprehensive benchmark containing 32,424 entities from 172 countries. The dataset is categorized into "Notable People" (e.g., politicians, athletes) and "Things" (e.g., culturally significant foods, landmarks, animals). Entities were sourced from Wikipedia and verified to ensure a clear primary country of origin.
Geographic Disparities in Success Rates
The results revealed a stark divide in performance based on geographic origin. Models were significantly more successful at deducing entities from the Global North and West compared to the Global South and East.
For instance, in the canonical 20-turn setting, Gemini 2.0 Flash achieved a 30.9% success rate for notable people from the Global North, but only 10.2% for those from the Global South. A similar gap was observed between the Global West (23.4%) and the Global East (10.6%). Generally, models were better at identifying people (25-36% success) than culturally specific objects (9-13%).
Key Insights: The "Western Default"
The study uncovered a "Western Default" in the models' reasoning processes. When attempting to identify non-Western entities, models often systematically eliminated Western possibilities first. For example, when guessing an Indian poet, a model might first ask if the person is American, British, or French before considering Asian origins.
Furthermore, the researchers found that language is not a "silver bullet." Playing the game in a local language (e.g., Hindi for an Indian entity) did not significantly bridge the performance gap. In some cases, models were more accurate identifying Asian entities when playing in English than in the region's native language.
Conclusion
The findings suggest that geographic bias in LLMs is deeper than simple training data frequency. Even when models possess the necessary information (as evidenced in unlimited turn settings), they are less efficient at accessing it for the Global South. Future work may involve using interpretability tools to pinpoint the neural layers responsible for these geographic "blind spots" and exploring more balanced training datasets.
Summary updated