Is Wikipedia succeeding in reducing gender bias? Assessing changes in gender bias in Wikipedia using word embeddings

Katja Geertruida Schmahl, Tom J. Viering, Stavros Makrodimitris, Arman Naseri Jahfari, David M. J. Tax, Marco Loog2020 · DOI

Abstract

Katja Geertruida Schmahl, Tom Julian Viering, Stavros Makrodimitris, Arman Naseri Jahfari, David Tax, Marco Loog. Proceedings of the Fourth Workshop on Natural Language Processing and Computational Social Science.

2020.

Introduction

Word embeddings—mathematical representations of words used in modern NLP—often inherit societal biases from their training corpora. For instance, if a model frequently encounters "doctor" near male pronouns, it learns a "male" association, potentially leading to discriminatory outcomes in applications like search rankings or machine translation.

While historical research shows gender bias in literature decreased over the 20th century, Wikipedia offers a unique "living" corpus. Since approximately 2013, editors have actively worked to bridge the gender gap. This paper investigates whether these manual editorial interventions are successfully shifting the latent gender biases within Wikipedia's word embeddings.

Methodology

The researchers conducted a temporal analysis using Wikipedia snapshots from 2006 to 2020. They trained Word2Vec (Continuous Bag-of-Words) models on each snapshot, retaining common gendered words to capture associations.

The primary metric used was the Word Embedding Association Test (WEAT). WEAT measures the relative cosine similarity between category word sets (e.g., Science: physics, nasa, experiment) and attribute word sets (Male: man, son, he vs. Female: woman, daughter, she). To ensure observed changes were statistically significant and not merely random noise, the authors compared the "slope" of bias change in target categories against the shifts of 1,000 sets of random words.

Datasets

  • Wikipedia Dumps: Full backups of articles across 11 years.
  • "New Articles" Subset: Articles added in the two years preceding a snapshot, used to determine if contemporary writing is more neutral than the "legacy" corpus.
  • WordSim353: A benchmark to validate that the embeddings accurately captured word semantics.

Key Results

The analysis revealed distinct trends across different categories. While some areas showed movement toward neutrality, others remained stagnant or even diverged.

Gender bias over time for all vs. new Wikipedia articles.
  • Career: This category remained the most biased, showing a strong, consistent association with male terms that did not significantly change over the 14-year period.
  • Science & Family: Both categories showed a trend toward stereotypical neutrality (Science moving from male-biased toward neutral; Family moving from female-biased toward neutral). However, these changes were not statistically significant when compared to random word sets, suggesting the "improvement" might be a byproduct of general embedding space drift.
  • New Articles vs. Full Corpus: Embeddings from newly added articles showed steeper improvement slopes for Career and Family. Conversely, these "new" articles were actually more biased in Science and Arts than the older corpus, indicating that new content is not a universal fix for neutrality.

Key Insights and Discussion

The "Arts" Divergence

Unlike other categories, "Arts" (e.g., poetry, dance, drama) showed a statistically significant trend toward becoming more female-biased. This may be linked to a high volume of new biographies for female artists.

Percentage of female biographies across different occupations (2017-2020).

The Lexical Bias Trap

A subtle issue identified is that efforts to fix coverage bias can inadvertently reinforce linguistic bias. When editors add articles about women, they often use gender-specific qualifiers like "female scientist." This strengthens the association between "female" and "scientist" in the model, while the generic term "scientist" remains tethered to "male."

Random Drift

One of the study's critical findings is the role of random drift. By using a random word baseline, the authors discovered that many "improvements" in bias scores were indistinguishable from the background noise of the vocabulary's shifting associations over time.

Limitations and Future Work

The study notes a "blind spot" between 2011 and 2013 due to missing Wikipedia backups. Additionally, highly influential figures like Einstein or Shakespeare might disproportionately influence the bias scores of their respective categories. Future research could utilize more granular article analysis—distinguishing between biographies and conceptual articles—to pinpoint the specific drivers of these latent biases.

Summary updated

BibTeX
@misc{schmahl2020is,
  author = {Katja Geertruida Schmahl and Tom J. Viering and Stavros Makrodimitris and Arman Naseri Jahfari and David M. J. Tax and Marco Loog},
  title = {{Is Wikipedia succeeding in reducing gender bias? Assessing changes in gender bias in Wikipedia using word embeddings}},
  year = {2020},
  doi = {10.18653/v1/2020.nlpcss-1.11},
}
Figure