Exploring Human Gender Stereotypes with Word Association Test

Yupei Du, Yuanbin Wu, Man LanEMNLP 2019 · OpenReview

Abstract

Traditional methods for quantifying gender bias in AI rely on word co-occurrences in large text corpora. However, these methods often fail to capture the nuanced mental associations held by humans. This paper introduces a novel approach using Word Association Tests (WAT) to build a graph-based model of stereotypes. By propagating gender information through a human-generated association network, the researchers demonstrate that WAT-based bias scores align more closely with human judgment than standard word embeddings and reveal that current de-biasing techniques may be insufficient.

Introduction

The study of gender bias in Natural Language Processing (NLP) has primarily focused on word embeddings like Word2Vec or GloVe. These models quantify bias by measuring the proximity of words (e.g., "engineer") to gendered terms (e.g., "he"). While effective, this approach is limited by its reliance on textual co-occurrence, which does not always reflect the direct cognitive associations of the human mind. Furthermore, validating these biases is difficult for abstract concepts where objective statistical "ground truth" (like census data for occupations) is unavailable.

Methodology: Stereotype Propagation

To address these gaps, the authors utilize the Small World of Words English (SWOWEN) dataset, a large-scale Word Association Test containing responses from over 90,000 participants. They construct a Word Association Graph where nodes are words and edges represent the strength of human associations.

The core of their analysis is "Stereotype Propagation." Starting with a small set of gender-specific seed pairs (e.g., man/woman, father/mother), they apply a Label Propagation algorithm. This allows gender information to flow across the graph until it reaches a stationary state, assigning a bias score to over 12,000 words.

Visualizing the propagation of gender stereotypes from seed words to attributes and occupations.

This graph-based approach captures indirect links that co-occurrence models miss. For instance, the association between "men" and "buildings" is mediated by the shared attribute of being "tall," a connection readily apparent in human associations but often obscured in raw text.

Results and Validation

The WAT-based model demonstrates superior performance in capturing human stereotypes. When compared against manual annotations of profession stereotypes, the model achieved a Pearson correlation of 0.66, significantly outperforming standard word embedding methods which typically score around 0.57.

Distribution of gender bias scores for selected words, showing clear separation between masculine and feminine associations.

Validation against U.S. Census data and Implicit Association Test (IAT) concepts further confirmed the model's accuracy. It successfully identified social biases, such as the masculine association of "science" and "career" versus the feminine association of "arts" and "family," with high effect sizes (Cohen’s $d$ up to 1.55).

The Persistence of Bias

A critical finding of the study is the inadequacy of current "de-biasing" algorithms. When the authors tested methods like "Hard-Debias" on word embeddings, they found that while the resulting vectors appeared neutral, the underlying associations within the WAT graph remained strongly biased. This suggests that stereotypes are deeply embedded in the structural relationships of language, and current methods may only be masking these biases at the surface level.

Conclusion

By shifting the focus from text co-occurrence to human word associations, this research provides a more authentic and robust framework for quantifying gender stereotypes. The Word Association Test reveals a "small world" of interconnected biases that are more pervasive and harder to remove than previously thought, highlighting the need for more sophisticated approaches to fairness in AI.

Summary updated

BibTeX
@inproceedings{du2019exploring,
  author = {Yupei Du and Yuanbin Wu and Man Lan},
  title = {{Exploring Human Gender Stereotypes with Word Association Test}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=rO-blpL00f9},
}
Figure