To evaluate whether a language model or word embedding contains social bias, researchers typically construct a metric based on a curated set of words or sentence templates. For instance, we might measure how close the vector for programmer is to man versus woman, or evaluate whether a language model predicts he or she with higher probability in the template "The [occupation] walked into the room."
However, a growing body of literature reveals that these measurement tools are surprisingly fragile. Instead of acting as stable rulers, our bias metrics behave like rubber bands: minor, meaning-preserving changes to the evaluation vocabulary or prompt structure can cause bias scores to fluctuate wildly, drop to zero, or even flip directions entirely. This phenomenon—driven by template and lexical sensitivity—suggests that many benchmarks measure the specific linguistic quirks of the evaluation setup rather than a model's underlying social associations.
The Geometry of Reference Pairs
In static word embeddings, a common way to quantify bias is to define a "gender axis" by subtracting the vector of one word from another (e.g., $\vec{v}_{\text{man}} - \vec{v}_{\text{woman}}$). The bias of a target word, like programmer, is then measured by its projection onto this axis.
The underlying assumption is that this vector difference cleanly isolates the semantic concept of gender. However, in Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs, Alison Sneyd et al. demonstrate that this assumption is mathematically fragile. Word vectors do not just capture abstract concepts; they encode a complex mixture of word frequency, grammatical roles, and specific lexical connotations.
When you swap the reference pair from man/woman to he/she, the orientation of the axis shifts. The authors show that this minor swap can significantly alter or even reverse the measured bias of target terms. This suggests that famous embedding analogies (such as "man is to computer programmer as woman is to homemaker") may be artifacts of local vector geometry and word similarity rather than robust reflections of societal stereotypes.
Lexical Sampling Variance in Seed Sets
When moving beyond single pairs to broader categories—such as tracking historical stereotypes over a century (Nikhil Garg et al. in Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes), analyzing biomedical literature (Anthony Rios et al. in Quantifying 60 Years of Gender Bias in Biomedical Research with Word Embeddings), or measuring offensive language (Fatma Elsafoury et al. in SOS: Systematic Offensive Stereotyping Bias in Word Embeddings)—researchers rely on curated lists of words known as "seed sets."
The intuition behind using a seed set is that averaging across multiple words will smooth out individual lexical quirks. But as Maria Antoniak et al. point out in Bad Seeds: Evaluating Lexical Methods for Bias Measurement, seed sets are not neutral instruments. They carry their own embedded cultural biases and linguistic features, such as word frequency imbalances, which heavily skew the output of bias detection algorithms.
In a reproducibility study, Jille Van der Togt et al. confirmed in Badder Seeds: Reproducing the Evaluation of Lexical Methodsfor Bias Measurement that factors like word ordering, pairing, and internal semantic similarity within seed sets fundamentally alter the resulting bias subspace. Consequently, small changes in word composition lead to highly unstable bias scores (Maximilian Spliethöver et al. in Bias Silhouette Analysis: Towards Assessing the Quality of Bias Metrics for Word Embedding Models).
As Nikil Roashan Selvam et al. demonstrate in The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks, subsampling different sets of occupations can produce overlapping score distributions across models. This high variance means that a single bias score fails to generalize across the broader domain it is meant to represent, making it difficult to reliably rank which model is "less biased."
This sensitivity also extends to cross-corpus polarization metrics, where ambiguous or poorly selected topical wordsets yield misleading results (Federico Bianchi et al. in SWEAT: Scoring Polarization of Topics across Different Corpora), and to semantic alignment tasks where the choice of anchor words completely alters the interpreted linguistic shift (Maurício Gruppi et al. in SenSE: A Toolkit for Semantic Change Exploration via Word Embedding Alignment).
Contextual Noise in Template Probing
For modern pre-trained and masked language models, bias is often measured using fill-in-the-blank sentence templates. The goal is to isolate word-level associations by placing target words into semantically neutral contexts.
However, language models are highly sensitive to local context and superficial linguistic cues. In Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models, Pieter Delobelle et al. show that supposedly neutral templates fail to isolate word-level associations from contextual noise. Minor lexical variations—like changing a single verb or conjunction—can drastically alter or flip a model's fairness score.
Why does this happen? Sarah Alnegheimish et al. argue in Using Natural Sentence Prompts for Understanding Biases in Language Models that synthetic, rigid templates often introduce their own strong connotations. When a model evaluates a synthetic prompt, it may be reacting to the artificial framing of the template rather than demonstrating an underlying social stereotype.
When the authors replaced synthetic templates with natural sentences harvested from Wikipedia, the richer context significantly reduced the observed bias, revealing that previous metrics were largely capturing "factory setting" priors of the template text itself. Similarly, Bum Chul Kwon et al. found in An Empirical Study on Pseudo-log-likelihood Bias Measures for Masked Language Models Using Paraphrased Sentences that pseudo-log-likelihood bias measures are overly sensitive to superficial wording rather than the underlying semantic meaning of the sentence.
Overcoming the Subjectivity of Curated Lists
Because manually curated seed words introduce researcher subjectivity and lack scalability across different cultural or linguistic contexts (Haozhe An et al. in Learning Bias-reduced Word Embeddings Using Dictionary Definitions), the field is shifting toward more robust evaluation and mitigation strategies. Recognizing that our "rulers" are inherently unstable is the first step toward building more reliable benchmarks that capture genuine model beliefs rather than template-specific quirks.
Key Foundational Work
Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs (2020) — contributed 3 concept elements to this direction.
Word embeddings quantify 100 years of gender and ethnic stereotypes (2017) — contributed 1 concept elements to this direction.
Quantifying 60 Years of Gender Bias in Biomedical Research with Word Embeddings (2020) — contributed 1 concept elements to this direction.