Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics

Aylin Caliskan, Pimparkar Parth Ajay, Tessa Charlesworth, Robert Wolfe, Mahzarin R. Banaji

Abstract

The statistical regularities in language corpora encode well-known social biases into word embeddings. Here, we focus on gender to provide a comprehensive analysis of group-based biases in widely-used static English word embeddings trained on internet corpora (GloVe 2014, fastText 2017). Using the Single-Category Word Embedding Association Test, we demonstrate the widespread prevalence of gender biases that also show differences in: (1) frequencies of words associated with men versus women; (b) part-of-speech tags in gender-associated words; (c) semantic categories in gender-associated words; and (d) valence, arousal, and dominance in gender-associated words. First, in terms of word frequency: we find that, of the 1,000 most frequent words in the vocabulary, 77% are more associated with men than women, providing direct evidence of a masculine default in the everyday language of the English-speaking world. Second, turning to parts-of-speech: the top male-associated words are typically verbs (e.g., fight, overpower) while the top female-associated words are typically adjectives and adverbs (e.g., giving, emotionally). Gender biases in embeddings also permeate parts-of-speech. Third, for semantic categories: bottom-up, cluster analyses of the top 1,000 words associated with each gender. The top male-associated concepts include roles and domains of big tech, engineering, religion, sports, and violence; in contrast, the top female-associated concepts are less focused on roles, including, instead, female-specific slurs and sexual content, as well as appearance and kitchen terms. Fourth, using human ratings of word valence, arousal, and dominance from a ~20,000 word lexicon, we find that male-associated words are higher on arousal and dominance, while female-associated words are higher on valence.

Introduction

Gender bias in artificial intelligence is often discussed through the lens of specific, high-profile stereotypes—such as the association of "doctor" with men and "nurse" with women. However, research suggests that bias is not merely a collection of isolated associations but is baked into the very structural architecture of how machines process language. This paper investigates how gender bias permeates word embeddings—the numeric vector representations that serve as the foundation for modern natural language processing (NLP).

By analyzing popular static embeddings like GloVe and fastText, this study reveals a pervasive "masculine default." This bias manifests across four critical dimensions: word frequency, syntax (parts of speech), semantic clustering, and psycholinguistic dimensions of meaning.

Methodology

The researchers evaluated two primary models: GloVe (trained on 840 billion tokens) and fastText (trained on 600 billion tokens). To quantify bias, they employed the Single-Category Word Embedding Association Test (SC-WEAT). Similar to the Implicit Association Test (IAT) used in social psychology, SC-WEAT measures the relative distance of target words (e.g., "ambition") to male-associated attribute words (e.g., "he," "man") versus female-associated attribute words (e.g., "she," "woman") within the vector space.

To provide a multi-dimensional view of this bias, the authors integrated several datasets:

  • Wordfreq: To analyze frequency distributions across the English language.
  • NRC-VAD Lexicon: A dataset of ~20,000 words rated by humans for Valence (positivity), Arousal (intensity), and Dominance (control).
  • Unsupervised K-means Clustering: To identify semantic themes without human-defined categories.
  • Part-of-Speech (POS) Tagging: To determine the grammatical roles associated with each gender.

The Masculine Default: Frequency and Syntax

One of the most striking findings is the existence of a "masculine default" in everyday language. Among the 1,000 most frequent words in the vocabulary, 77% are more closely associated with men than women. This skew is even more pronounced at the extreme: in the GloVe model, 93% of the 100 most frequent words are male-associated. This suggests that for an AI model, the "standard" or "default" human experience is represented through a male lens.

This bias extends to the grammatical structure of language. The study found a clear syntactic divide:

  • Male-associated words are significantly more likely to be verbs, representing agentic actions (e.g., "fight," "overpower," "lead").
  • Female-associated words are more likely to be adjectives and adverbs, representing descriptive or passive states (e.g., "beautiful," "emotionally," "giving").

Semantic Landscapes and the "Women are Wonderful" Effect

Using unsupervised clustering, the researchers identified 11 distinct semantic clusters for each gender. The male clusters were defined by professional and societal roles, including "Engineering and Automotive," "Big Tech," "Sports," and "Religion."

In contrast, the female clusters were less focused on roles and more on descriptions and domesticity, including "Beauty and Appearance" and "Cooking and Kitchen." Most disturbingly, the female semantic space contained a high concentration of sexual profanities and obscene adult content, reflecting the biases inherent in the internet-crawled corpora used to train these models.

Psycholinguistically, the data supports the "Women are Wonderful" effect—a phenomenon where women are associated with high valence (positivity/niceness) but low dominance and arousal. Conversely, male-associated words correlated strongly with high dominance and arousal, reinforcing the stereotype of men as powerful and active agents.

Case Study: Big Tech

To examine industry-specific bias, the authors tested a list of 965 words associated with major technology companies. They found that 62% of these terms were more closely associated with men. This indicates that the "bro-culture" often cited in the tech industry is mathematically encoded into the tools that the industry itself produces, potentially creating a feedback loop of exclusion.

Conclusion and Limitations

This comprehensive analysis demonstrates that gender bias in word embeddings is structural, affecting how models perceive frequency, agency, and social roles. While this study focuses on static embeddings (GloVe and fastText), the findings provide a vital baseline for understanding how bias enters AI systems.

The authors acknowledge two primary limitations: the focus on static rather than contextual embeddings (like BERT or GPT) and the reliance on a binary representation of gender. Future research must address how these biases shift in context and how to better represent non-binary and fluid identities that are currently erased or underrepresented in massive training datasets.

Summary updated

Abstract

The statistical regularities in language corpora encode well-known social biases into word embeddings. Here, we focus on gender to provide a comprehensive analysis of group-based biases in widely-used static English word embeddings trained on internet corpora (GloVe 2014, fastText 2017). Using the Single-Category Word Embedding Association Test, we demonstrate the widespread prevalence of gender biases that also show differences in: (1) frequencies of words associated with men versus women; (b) part-of-speech tags in gender-associated words; (c) semantic categories in gender-associated words; and (d) valence, arousal, and dominance in gender-associated words. First, in terms of word frequency: we find that, of the 1,000 most frequent words in the vocabulary, 77% are more associated with men than women, providing direct evidence of a masculine default in the everyday language of the English-speaking world. Second, turning to parts-of-speech: the top male-associated words are typically verbs (e.g., fight, overpower) while the top female-associated words are typically adjectives and adverbs (e.g., giving, emotionally). Gender biases in embeddings also permeate parts-of-speech. Third, for semantic categories: bottom-up, cluster analyses of the top 1,000 words associated with each gender. The top male-associated concepts include roles and domains of big tech, engineering, religion, sports, and violence; in contrast, the top female-associated concepts are less focused on roles, including, instead, female-specific slurs and sexual content, as well as appearance and kitchen terms. Fourth, using human ratings of word valence, arousal, and dominance from a ~20,000 word lexicon, we find that male-associated words are higher on arousal and dominance, while female-associated words are higher on valence.
BibTeX
@inproceedings{caliskan2022gender,
  author = {Aylin Caliskan and Pimparkar Parth Ajay and Tessa Charlesworth and Robert Wolfe and Mahzarin R. Banaji},
  title = {{Gender Bias in Word Embeddings: A Comprehensive Analysis of Frequency, Syntax, and Semantics}},
  booktitle = {AIES},
  year = {2022},
  url = {https://openreview.net/forum?id=ny35c-V_4k0},
}
Figure