Dataset Geography: Mapping Language Data to Language Users

Fahim Faisal, Yinkai Wang, Antonios Anastasopoulos

Introduction

NLP datasets are often assumed to be neutral, but they carry deep geographic and cultural biases. This research introduces a method to map the "geography" of a dataset by linking its entities to specific countries. The findings reveal that most benchmarks are heavily skewed toward wealthy, Western nations, regardless of the language being studied.

As NLP expands to support thousands of languages, a critical question arises: Does the content of our data actually reflect the lives and interests of the people who speak those languages? Currently, many multilingual datasets are created by translating English benchmarks or scraping Wikipedia, often resulting in a "cultural mismatch." For example, a Swahili dataset might primarily contain mentions of entities in the US and Europe rather than Tanzania or Kenya. The authors argue that a model’s performance on such "Western-centric" data is a poor proxy for its real-world utility for native speakers.

Methodology: Mapping Dataset Geography

The authors propose a three-step pipeline to visualize "Dataset Geography":

  1. Entity Recognition: Identify named entities (people, places, organizations) within a dataset.
  2. Entity Linking: Map these entities to unique Wikidata IDs.
  3. Geographic Mapping: Use Wikidata metadata (place of birth, citizenship, headquarters) to associate entities with specific countries.

A key technical contribution is the "NER-Relaxed" approach. Traditional entity linking requires a high-quality Named Entity Recognition (NER) model to find spans first—a luxury not available for many low-resource languages. The authors found they could bypass explicit NER by simply wrapping an entire sentence in a "start/end" span and feeding it to a multilingual autoregressive linker (mGENRE). This approach achieved 60-85% of the accuracy of NER-informed models, making geographic analysis viable for under-served languages.

Key Insights: The Economic and Structural Drivers of Bias

The study evaluates several prominent NLP benchmarks, including WikiANN, MasakhaNER, SQuAD, and TyDi-QA. Several patterns emerge:

  • GDP as a Predictor: Across almost all datasets, the number of mentions a country receives correlates more strongly with its Gross Domestic Product (GDP) than with its population or geographic proximity to the language's speakers. Essentially, datasets talk about where the money is, not where the people are.
  • The "Translation Trap": Datasets created by translating English text (like MLQA) are significantly less representative of local cultures than those built from scratch by local communities (like TyDi-QA or MasakhaNER).
  • Participatory Design: MasakhaNER, created through a participatory approach involving native speakers, showed much higher "in-country" representativeness for languages like Wolof and Yoruba compared to the silver-standard WikiANN.
Comparison of geographic representation across different datasets. SQuAD (top left) shows a heavy concentration in the US and Europe, while TyDi-QA (top right) demonstrates the impact of dataset construction methods on geographic diversity.

Case Study: Swahili and the Representativeness Gap

The geographic skew is particularly evident in low-resource language datasets. In the Swahili portion of MasakhaNER, only about 17% of entity mentions related to the primary Swahili-speaking region (East Africa), while the USA and Western Europe were significantly over-represented.

Dataset Entities Map for Masakhaner Swahili, showing that despite being a Swahili dataset, the USA is the most represented country, with only 16.97% of entities originating from the primary Swahili-speaking region.

This lack of representation has tangible effects on model fairness. The authors found that models trained on Western-centric data (like SQuAD) perform significantly worse on test samples containing Asian or African entities. For Telugu, a SQuAD-trained model was 20 percentage points worse on Asian-related questions than a model trained on locally-sourced data.

Limitations and Future Outlook

While this mapping provides a powerful diagnostic tool, it relies on Wikidata, which itself suffers from Western-centric bias, potentially underestimating the representativeness of low-resource regions. Furthermore, mapping to "countries" is a coarse measure that misses sub-national cultural nuances.

Future work involves releasing the tool as a Python package and expanding the analysis to more tasks and finer-grained geographic clusters. The goal is to move toward NLP data that truly reflects the global diversity of its users.

Summary updated

Abstract

As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions. Code and data are available here: https://github.com/ffaisal93/dataset_geography. Additional visualizations are available here: https://nlp.cs.gmu.edu/project/datasetmaps/.
BibTeX
@inproceedings{faisal2022dataset,
  author = {Fahim Faisal and Yinkai Wang and Antonios Anastasopoulos},
  title = {{Dataset Geography: Mapping Language Data to Language Users}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Iag0GrYKoMv},
}
Figure