Wikidata as a Source of Demographic Information

Samir Abdaljalil, Hamdy MubarakACL 2024

TL;DR

This paper explores using individual names as a rich source of demographic information, demonstrating that machine learning models can predict country, gender, and language with reasonable accuracy. They leverage Wikidata to build a large, normalized dataset and find that a linear SVM approach outperforms more complex Transformer models for this task.

What Problem Does This Paper Address?

Understanding an individual's demographic attributes (like country, gender, and language) is crucial for various applications, from personalized user experiences to linguistic and social studies. Traditionally, this information is gathered through explicit user input, which can be tedious and lead to incomplete data. While names are known to carry demographic signals, systematically extracting this information at scale, especially across multiple languages and diverse regions, has been a challenge. Existing research often focuses on limited geographic areas or specific attributes, leaving a gap in comprehensive, multilingual demographic inference from names.

Methods

The authors employ a supervised classification approach. They experiment with two main types of models:

  • Linear Support Vector Machine (SVM): This standard machine learning algorithm is used to classify names based on character n-grams. The authors explore different n-gram ranges to capture patterns within names.
  • Transformer Models: They investigate two Transformer-based approaches:
  • Masked Language Modeling (MLM) Prediction: Using pre-trained BERT models, they frame the task as predicting masked tokens in carefully crafted sentence templates that include the person's name.
  • BERT Model Fine-tuning: They also fine-tune a BERT model specifically for the country prediction task.

Datasets

The primary dataset used in this research is derived from Wikidata. The authors extracted over 9 million unique entities, normalizing and cleaning the data to create a comprehensive dataset of over 1 million entities with associated country, gender, language, and region information. This dataset covers 100 countries, 17 regions, 48 languages, and 2 genders, with names in both Arabic and English.

For evaluation, they also used an external dataset of 1,500 names from 15 countries extracted from Twitter, aiming to test the models on less popular and more diverse names.

Key Insights and Ideas

  • Names are surprisingly informative: Even without explicit demographic data, names alone can provide significant clues about an individual's country, gender, and language.
  • Wikidata is a valuable resource: The paper highlights Wikidata's potential as a rich, albeit noisy, source for building large-scale demographic datasets, especially when combined with careful data curation and normalization.
  • Simplicity can be effective: Counterintuitively, a linear SVM model, when properly feature-engineered (using character n-grams), outperformed more complex Transformer models (like BERT) in terms of both accuracy and computational efficiency for this specific task.
  • Data normalization is critical: The raw data from Wikidata contains many variations in country names and transcription formats, necessitating extensive cleaning and normalization to create a usable dataset.
  • Regional information adds value: While predicting exact country can be challenging due to name similarities across borders, inferring a broader region can be more robust and useful for certain applications.

Key Results

  • Gender prediction: Achieved a weighted F1-score of 0.96 using the Linear SVM.
  • Language prediction: Achieved a weighted F1-score of 0.75 using the Linear SVM.
  • Region prediction: Achieved a weighted F1-score of 0.65 using the Linear SVM.
  • Country prediction: Achieved a weighted F1-score of 0.53 using the Linear SVM.
  • Transformer models: While competitive, BERT-based approaches generally yielded lower weighted F1-scores compared to the Linear SVM, particularly for country prediction (0.54 for fine-tuned BERT vs. 0.53 for SVM, but with significantly higher computational cost). Masked language modeling approaches showed lower performance across the board.
  • Twitter dataset validation: The SVM_Country model achieved a weighted F1-score of 0.54 on the Twitter dataset, confirming the robustness and generalizability of the models.

Limitations and Future Work

  • Limited Language Coverage: The current work focuses primarily on English and Arabic. Expanding to more languages would improve generalizability.
  • Data Biases: The models are trained on data from Wikidata, which may under-represent certain regions and ethnic groups, potentially affecting prediction capabilities for less common names. The "digital divide" can also impact data availability.
  • Potential Misuse: The ability to infer demographic information from names raises ethical concerns about potential misuse for discriminatory practices, similar to documented biases in hiring.
  • Ethnicity and Religion: While the paper mentions these as potential attributes, they were not the primary focus of the experiments. Future work could explore inferring these attributes.
  • More Granular Data: Exploring finer-grained geographical information beyond the 18 regions or incorporating other Wikidata attributes could further enhance predictions.
  • Computational Cost: Transformer models, while powerful, are computationally expensive. Future work could focus on optimizing these models or finding more efficient ways to leverage their capabilities.

Summary updated

BibTeX
@inproceedings{abdaljalil2024wikidata,
  author = {Samir Abdaljalil and Hamdy Mubarak},
  title = {{Wikidata as a Source of Demographic Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure