Introduction
NLP datasets are often assumed to be neutral, but they carry deep geographic and cultural biases. This research introduces a method to map the "geography" of a dataset by linking its entities to specific countries. The findings reveal that most benchmarks are heavily skewed toward wealthy, Western nations, regardless of the language being studied.
As NLP expands to support thousands of languages, a critical question arises: Does the content of our data actually reflect the lives and interests of the people who speak those languages? Currently, many multilingual datasets are created by translating English benchmarks or scraping Wikipedia, often resulting in a "cultural mismatch." For example, a Swahili dataset might primarily contain mentions of entities in the US and Europe rather than Tanzania or Kenya. The authors argue that a model’s performance on such "Western-centric" data is a poor proxy for its real-world utility for native speakers.
Methodology: Mapping Dataset Geography
The authors propose a three-step pipeline to visualize "Dataset Geography":
- Entity Recognition: Identify named entities (people, places, organizations) within a dataset.
- Entity Linking: Map these entities to unique Wikidata IDs.
- Geographic Mapping: Use Wikidata metadata (place of birth, citizenship, headquarters) to associate entities with specific countries.
A key technical contribution is the "NER-Relaxed" approach. Traditional entity linking requires a high-quality Named Entity Recognition (NER) model to find spans first—a luxury not available for many low-resource languages. The authors found they could bypass explicit NER by simply wrapping an entire sentence in a "start/end" span and feeding it to a multilingual autoregressive linker (mGENRE). This approach achieved 60-85% of the accuracy of NER-informed models, making geographic analysis viable for under-served languages.
Key Insights: The Economic and Structural Drivers of Bias
The study evaluates several prominent NLP benchmarks, including WikiANN, MasakhaNER, SQuAD, and TyDi-QA. Several patterns emerge:
- GDP as a Predictor: Across almost all datasets, the number of mentions a country receives correlates more strongly with its Gross Domestic Product (GDP) than with its population or geographic proximity to the language's speakers. Essentially, datasets talk about where the money is, not where the people are.
- The "Translation Trap": Datasets created by translating English text (like MLQA) are significantly less representative of local cultures than those built from scratch by local communities (like TyDi-QA or MasakhaNER).
- Participatory Design: MasakhaNER, created through a participatory approach involving native speakers, showed much higher "in-country" representativeness for languages like Wolof and Yoruba compared to the silver-standard WikiANN.
Case Study: Swahili and the Representativeness Gap
The geographic skew is particularly evident in low-resource language datasets. In the Swahili portion of MasakhaNER, only about 17% of entity mentions related to the primary Swahili-speaking region (East Africa), while the USA and Western Europe were significantly over-represented.
This lack of representation has tangible effects on model fairness. The authors found that models trained on Western-centric data (like SQuAD) perform significantly worse on test samples containing Asian or African entities. For Telugu, a SQuAD-trained model was 20 percentage points worse on Asian-related questions than a model trained on locally-sourced data.
Limitations and Future Outlook
While this mapping provides a powerful diagnostic tool, it relies on Wikidata, which itself suffers from Western-centric bias, potentially underestimating the representativeness of low-resource regions. Furthermore, mapping to "countries" is a coarse measure that misses sub-national cultural nuances.
Future work involves releasing the tool as a Python package and expanding the analysis to more tasks and finer-grained geographic clusters. The goal is to move toward NLP data that truly reflects the global diversity of its users.
Summary updated