Abstract
Social bias benchmarks are increasingly used to evaluate large language models (LLMs), yet they often rely on rigid, templated sentence structures. This paper demonstrates that these benchmarks are surprisingly brittle: minor, meaning-preserving modifications—such as synonym substitution or adding neutral adjectives—can drastically alter a model's "bias score" and even reorder model rankings. These findings suggest that current benchmarks may be capturing dataset construction artifacts and non-social model quirks, such as positional bias, rather than genuine social prejudice.
Introduction
As large language models (LLMs) become ubiquitous, researchers have developed benchmarks like WINOGENDER and BIASNLI to quantify social biases, such as the tendency to associate specific occupations with a particular gender. These benchmarks typically use templated sentences to observe if a model's prediction changes based on gendered pronouns or occupation labels. However, this study investigates whether these benchmarks are "the tail wagging the dog"—where arbitrary choices in dataset construction, rather than underlying social bias, drive the results.
Methodology: Perturbing the Benchmarks
To test the robustness of existing social bias metrics, the researchers created "alternate constructions" by applying several shallow modifications to standard templates. These changes maintain the core meaning and the social bias being tested but alter the linguistic structure:
- Synonym Substitution: Replacing non-pivotal words (e.g., "warned" → "cautioned").
- Adding Descriptors: Inserting neutral adjectives (e.g., "the cruel engineer").
- Adding Clauses: Inserting phrases that increase the distance between entities (e.g., "who just returned from the beach").
- Negation: Changing a verb to its negative form (e.g., "bought" → "did not buy").
- Random Subsampling: Evaluating how results vary when using different subsets of the occupation word list.
Experimental Setup
The study evaluated several models, including SpanBERT, UnifiedQA, Longformer, ALBERT, and RoBERTa, across two primary benchmarks:
- WINOGENDER: A coreference resolution task linking pronouns to occupations or participants.
- BIASNLI: A Natural Language Inference (NLI) task checking for stereotypical inferences.
Results: The Fragility of Bias Metrics
The most significant finding is the extreme instability of model rankings and scores across different perturbations. A model that appears "fairer" on a baseline benchmark may perform significantly worse when a simple adjective is added or a verb is negated.
Key Findings:
- Massive Score Swings: On the BIASNLI benchmark, negating verbs caused the measured bias for an ELMo-based model to drop from 41.64 to 13.40.
- Winogender Volatility: Adding an adjective after an occupation reduced the measured bias of the UnifiedQA (large) model by 10% compared to the baseline.
- Confounding Non-Social Biases: Benchmarks often mistake "positional bias" (the tendency to link a pronoun to the nearest noun) for gender bias.
- Ranking Instability: Model rankings are not consistent across templates, making these benchmarks unreliable for definitive model comparisons or deployment decisions.
Discussion and Future Directions
The study highlights that models often lack a deep understanding of the social concepts these benchmarks aim to measure, instead reacting to "shallow" linguistic triggers. Because these benchmarks rely on repetitive templates, they are highly susceptible to overfitting to specific structures. The authors acknowledge limitations, such as the focus on English and binary gender frameworks. Future work should incorporate syntactic and semantic diversity and "uncertainty measures" rather than a single score to ensure benchmarks are measuring what they claim to measure.
Summary updated