Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis

Scott Friedman, Sonja Schmer-Galunder, Anthony Chen, Jeffrey RyeACL 2019

TL;DR

This paper explores how gender biases embedded in language, specifically in word embeddings derived from Twitter data, can reflect and even predict real-world gender gaps across different cultures. By analyzing word embeddings from 99 countries and 51 US regions, the researchers found that certain thematic word sets' biases strongly correlate with specific gender gap statistics, offering a computational lens into societal gender disparities.

What Problem Does This Paper Address?

Modern natural language processing (NLP) models, trained on vast amounts of text data from sources like journalism and social media, often inherit and amplify societal biases, including those related to gender. While much research focuses on debiasing these models to improve their performance, this paper proposes a different angle: using these inherent biases as a valuable signal. The authors argue that word embedding biases can serve as a proxy for actual gender gaps in society, providing a large-scale, data-driven way to understand cultural attitudes and their manifestation in disparities in education, politics, economics, and health. They aim to quantify these biases and explore their relationship with measurable gender gaps across diverse cultural contexts.

Methods

The core of the paper involves two main steps:

  1. Quantifying Gender Bias in Word Embeddings: The researchers trained custom Word2Vec word embeddings for each of the 99 countries and 51 US regions using Twitter data. To measure gender bias within these embeddings, they computed an "average axis projection." This involves defining "male" and "female" vectors by averaging the embeddings of corresponding words (e.g., "man," "woman," "he," "she") and then projecting a neutral word set (like "government" terms) onto the axis defined by the difference between the male and female vectors. A higher projection value indicates a stronger association of that neutral concept with the "female" direction.
  2. Correlating Bias with Gender Gaps: They then correlated these computed gender biases with various statistical gender gap metrics. To do this, they created "themed word sets" representing concepts like government, threat, communal activities, criminal behavior, and childcare. For each themed word set, they calculated its gender bias across different regions/countries and then regressed this bias against corresponding gender gap statistics. This allowed them to assess the strength and direction of the relationship between language bias and societal disparities.

Datasets

  • Training Data: Public tweets from U.S. and international Twitter users collected over 100 days in 2018. Tweets were filtered for length, and URLs, usernames, hashtags, images, and emojis were replaced with tokens. Data was split into two main datasets: one for 51 U.S. regions (50 states + Washington D.C.) and another for 99 countries.
  • Gender Gap Statistics:
    • International: 18 gender gap metrics from the World Economic Forum's Global Gender Gap Index (GGGI).
  • U.S. Domestic: 5 statistics including exercise rates (CDC), wage gap and workforce data (U.S. Census Bureau), female percentages in math/computer science degrees (Society of Women Engineers), and female percentages of state legislators (Represent Women).

Key Insights and Ideas

  • Language Bias as a Societal Mirror: The most significant idea is that biases in language, as captured by word embeddings, are not just technical artifacts but can serve as a powerful indicator of real-world gender disparities.
  • Thematic Specificity Matters: Not all language biases correlate with all gender gaps. Biases in "government" words, for instance, strongly correlate with political empowerment gaps, while "threat" words show a connection to economic disparities like the gender pay gap. This suggests that specific cultural associations in language map onto specific societal issues.
  • Predictive Power of Language: The study demonstrates that word embedding biases can not only reflect but also predict statistical gender gaps, suggesting a dynamic relationship where language influences and is influenced by societal structures.
  • Cross-Cultural Regularities: The analysis across diverse countries and US states reveals some consistent patterns, indicating that certain linguistic associations with gender are widespread.
  • Valence and Dominance as Explanators: Adjectives correlated with increased gender gaps (e.g., lower pay equality) tended to have lower valence and dominance scores, suggesting that negative or less assertive language associations might be linked to greater societal disadvantages for women.

Key Results

  • Thematic word sets showed selective correlations with gender gap statistics. For example, the "government" word set was positively correlated with political empowerment gender gaps, and "threat" words showed an inverse correlation with the gender pay gap.
  • The study found that for the gender pay gap, adjectives whose female bias correlated with reduced pay gap (higher wages) had significantly higher valence and dominance scores than those correlated with increased pay gap (lower wages).
  • The Word2Vec algorithm with the axis projection bias metric performed best among the tested algorithms and metrics for correlating gender bias with gender gaps.
  • Randomly sampled word sets showed minimal correlation with gender gaps, underscoring the importance of thematic grouping.

Limitations and Future Work

  • English-Only Limitation: The use of English-only tweets limits the analysis to English-speaking populations and excludes valuable insights from other languages and cultures. This also means it doesn't capture voices of those with limited technology access or English proficiency.
  • Causality vs. Correlation: The study primarily establishes correlations, not causality. While the findings are consistent with social theories, they don't definitively prove that language bias causes gender gaps, or vice-versa.
  • Data Granularity: Some gender gaps, like "percentage of last 50 years with female head of state" or "survey-based wage equality," showed weaker correlations, suggesting they might have more complex relationships or require different analytical approaches.
  • Future Work: The authors suggest expanding to other languages, incorporating more diverse data sources (beyond just tweets), and using more advanced causal modeling techniques to better understand the interplay between language bias and societal gender gaps. They also note that blending tweet data with news or Wikipedia, as done in some prior work, might improve NLP accuracy but could dilute the very biases they are trying to study.

Summary updated

BibTeX
@inproceedings{friedman2019relating,
  author = {Scott Friedman and Sonja Schmer-Galunder and Anthony Chen and Jeffrey Rye},
  title = {{Relating Word Embedding Gender Biases to Gender Gaps: A Cross-Cultural Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure