Dataset Geography: Mapping Language Data to Language Users

Fahim Faisal, Yinkai Wang, Antonios Anastasopoulos

Abstract

As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions.

Introduction: The Cultural Disconnect in NLP

The expansion of NLP to support thousands of languages often prioritizes raw data availability over content representativeness. However, if a dataset for a specific language primarily discusses entities and contexts from Western nations, models trained on it may fail to serve the actual needs of that language's speakers. This paper introduces a method to quantify and visualize "geographical representativeness" by mapping mentioned entities to their countries of origin.

Quantifying Dataset Geography

The authors propose a three-step pipeline to map datasets to countries:

  1. Entity Identification: Extracting named entities (people, places, organizations).
  2. Entity Linking: Mapping these entities to unique Wikidata IDs.
  3. Geographic Mapping: Using Wikidata properties (e.g., "place of birth" or "headquarters location") to associate entities with specific countries.

To handle low-resource languages where high-quality NER models are unavailable, the authors introduce a "NER-Relaxed" approach. By using mGENRE, a multilingual autoregressive linker, they process entire sentences at once, bypassing the need for a dedicated NER step while maintaining the integrity of the resulting geographic maps.

The GDP Bias and Representativeness Gap

A central finding of this research is the "GDP Bias": a country's Gross Domestic Product is a much stronger predictor of its representation in a dataset than its population or geographic proximity to the language's speakers. Economic factors explain roughly 50-60% of the variance in country appearance.

Geographic distribution of entities in the MasakhaNER Swahili dataset, showing a heavy bias toward Western nations despite the language's East African roots.

Even datasets specifically designed for diversity exhibit this Western-centricity. For instance, in the Swahili portion of the MasakhaNER dataset, only 16.97% of entities relate to East African countries like Tanzania or Kenya, while the USA and Western Europe are significantly over-represented.

Benchmarking Representativeness: From SQuAD to NQ

The study evaluated several Question Answering (QA) and NER benchmarks, revealing significant disparities in "in-country" representation—the percentage of entities mentioned that originate from countries where the language is natively spoken.

  • Natural Questions (NQ): 80.07% in-country representation.
  • SQuAD: 62.73% in-country representation.
  • TyDi-QA: 57.33% in-country representation.
  • MLQA: 53.63% in-country representation.
The Natural Questions (NQ) dataset demonstrates high geographical representativeness, with 80% of entities originating from countries where English is spoken.

NQ, based on real Google search queries, emerged as the most representative dataset. This suggests that user-driven data collection is more effective at achieving cultural fairness than curator-driven methods. Furthermore, the study found that models trained on Western-biased data perform significantly worse on entities from the Global South, highlighting a tangible performance inequality.

Conclusion and Future Directions

While this mapping tool provides a vital audit mechanism, it faces limitations including Wikidata's own Western-centric bias and the inherent complexity of mapping languages strictly to national borders. Future work aims to increase granularity to sub-national regions and release the mapping tool as a Python package to help the community address geographic bias in NLP development.

Summary updated

Abstract

As language technologies become more ubiquitous, there are increasing efforts towards expanding the language diversity and coverage of natural language processing (NLP) systems. Arguably, the most important factor influencing the quality of modern NLP systems is data availability. In this work, we study the geographical representativeness of NLP datasets, aiming to quantify if and by how much do NLP datasets match the expected needs of the language speakers. In doing so, we use entity recognition and linking systems, also making important observations about their cross-lingual consistency and giving suggestions for more robust evaluation. Last, we explore some geographical and economic factors that may explain the observed dataset distributions. Code and data are available here: https://github.com/ffaisal93/dataset_geography. Additional visualizations are available here: https://nlp.cs.gmu.edu/project/datasetmaps/.
BibTeX
@inproceedings{faisal2022dataset,
  author = {Fahim Faisal and Yinkai Wang and Antonios Anastasopoulos},
  title = {{Dataset Geography: Mapping Language Data to Language Users}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Iag0GrYKoMv},
}
Figure