Abstract
Traditional methods for quantifying gender bias in AI rely on word co-occurrences in large text corpora. However, these methods often fail to capture the nuanced mental associations held by humans. This paper introduces a novel approach using Word Association Tests (WAT) to build a graph-based model of stereotypes. By propagating gender information through a human-generated association network, the researchers demonstrate that WAT-based bias scores align more closely with human judgment than standard word embeddings and reveal that current de-biasing techniques may be insufficient.
Introduction
The study of gender bias in Natural Language Processing (NLP) has primarily focused on word embeddings like Word2Vec or GloVe. These models quantify bias by measuring the proximity of words (e.g., "engineer") to gendered terms (e.g., "he"). While effective, this approach is limited by its reliance on textual co-occurrence, which does not always reflect the direct cognitive associations of the human mind. Furthermore, validating these biases is difficult for abstract concepts where objective statistical "ground truth" (like census data for occupations) is unavailable.
Methodology: Stereotype Propagation
To address these gaps, the authors utilize the Small World of Words English (SWOWEN) dataset, a large-scale Word Association Test containing responses from over 90,000 participants. They construct a Word Association Graph where nodes are words and edges represent the strength of human associations.
The core of their analysis is "Stereotype Propagation." Starting with a small set of gender-specific seed pairs (e.g., man/woman, father/mother), they apply a Label Propagation algorithm. This allows gender information to flow across the graph until it reaches a stationary state, assigning a bias score to over 12,000 words.
This graph-based approach captures indirect links that co-occurrence models miss. For instance, the association between "men" and "buildings" is mediated by the shared attribute of being "tall," a connection readily apparent in human associations but often obscured in raw text.
Results and Validation
The WAT-based model demonstrates superior performance in capturing human stereotypes. When compared against manual annotations of profession stereotypes, the model achieved a Pearson correlation of 0.66, significantly outperforming standard word embedding methods which typically score around 0.57.
Validation against U.S. Census data and Implicit Association Test (IAT) concepts further confirmed the model's accuracy. It successfully identified social biases, such as the masculine association of "science" and "career" versus the feminine association of "arts" and "family," with high effect sizes (Cohen’s $d$ up to 1.55).
The Persistence of Bias
A critical finding of the study is the inadequacy of current "de-biasing" algorithms. When the authors tested methods like "Hard-Debias" on word embeddings, they found that while the resulting vectors appeared neutral, the underlying associations within the WAT graph remained strongly biased. This suggests that stereotypes are deeply embedded in the structural relationships of language, and current methods may only be masking these biases at the surface level.
Conclusion
By shifting the focus from text co-occurrence to human word associations, this research provides a more authentic and robust framework for quantifying gender stereotypes. The Word Association Test reveals a "small world" of interconnected biases that are more pervasive and harder to remove than previously thought, highlighting the need for more sophisticated approaches to fairness in AI.
Summary updated