When a language model reads "The woman who hired the mechanic hurt ___self," English grammar fixes the answer: herself, because the reflexive must agree with the main-clause subject. The word mechanic sits inside a relative clause and is grammatically irrelevant. Any shift toward himself that its presence causes is therefore not a reflection of the training data's statistics about women. It is a stereotype leaking in from a word the sentence's structure has ruled out. This note proposes measuring that leakage directly, which we call stereotype attraction. It also proposes testing whether language models trained to predict along syntactic dependencies leak less than ordinary left-to-right models. A syntactic dependency is a grammatical link between a word and the word that governs it, such as a verb and its subject.
The opportunity is that grammar supplies a ground truth bias research usually lacks. Most bias benchmarks have to argue about what the "fair" output is. For off-structure influence the answer is clear: it should be zero, whatever one thinks about the inherited association between an occupation and a gender. Psycholinguistics already shows that sequential networks let linearly nearby nouns override structure for grammatical number. Bias research already shows that optimizers inflate the combined weight of many weak cues. Nobody has joined the two to separate inherited bias from structurally illegitimate amplification.
Core Claim
We claim that part of what is measured as bias amplification in language models is linear leakage: demographic associations of words that are close in the word sequence but structurally unrelated to the prediction site. We claim this component can be isolated with controlled minimal pairs, meaning sentence pairs that differ in one word. We further claim it is reduced, separately from the inherited component, by training objectives that make the model predict along dependency arcs.
The new element is not a new debiasing method. It is a decomposition of bias into an on-structure term and an off-structure term, together with a test of whether structural training moves them in different directions. That difference matters because of the result from StereoSet: a rank correlation of 0.87 between language-modeling score and stereotype score. A dependency objective that improves modeling might raise the on-structure term, consistent with StereoSet, while lowering the off-structure term. A single aggregate bias score would hide that.
Grounding in Prior Work
Two results motivate the proposal. Distinct patterns of syntactic agreement errors in recurrent networks and humans shows that LSTMs resolving subject-verb number agreement make errors that grow cumulatively with each intervening "attractor" noun. It also shows they err more across relative clauses than prepositional phrases, the reverse of humans, who attend only to the structurally closest attractor. Linzen and Leonard study only grammatical number; we carry their attractor design over to gender and other demographic attributes. Feature-Wise Bias Amplification shows that stochastic gradient descent overestimates many weak features associated with one class, amplifying bias even with balanced classes. Intervening context words are exactly such weak features. Together these predict that leakage should be cumulative in the number of attractors and larger for sequential models.
Two papers supply the method. Dependency-based Mixture Language Models (DMLM) adds an objective that predicts each token's future dependents. It mixes these syntactic expectations with attention, and it works on Transformers and GPT-2, improving Penn Treebank perplexity by over 20 points. We reuse this as the primary structural intervention. Its relations are unlabeled, so we also build a labeled variant that separates subject links from modifier links. Older structured models, which condition on exposed headwords (A Structured Language Model) or ancestral paths (Dependency Language Models for Sentence Completion), motivate the hypothesis that head-based conditioning should skip intervening modifiers. They are not practical at modern scale, so we use them only conceptually.
Several papers constrain the design. Is there Gender Bias in Dependency Parsing? showed that an apparent demographic effect in parsing was entirely explained by sentence length and dependency depth. Every comparison we make therefore holds length fixed, and we report effects as functions of tree distance rather than raw length. Directional Bias Amplification separates attribute-to-task bias from task-to-attribute bias, the latter meaning inferring a protected attribute from context. Stereotype attraction is a pure case of the latter, with the twist that syntax tells us which context is illegitimate. The paper also shows how threshold choices can masquerade as amplification, so we measure log-odds shifts rather than thresholded decisions. WinoBias already separates Type 2 sentences, where syntax determines coreference, from Type 1, where it does not. WinoBias evaluated coreference systems rather than language models, and it did not vary attractor position or count. We use it for external validity, not as the core probe.
The Diagnostic
We build templated minimal pairs in two families, varying one factor at a time while holding token length fixed with filler words.
In the grammatically determined family, the head is explicitly gendered ("The woman/man who [verb] the [occupation] ... hurt ___self"). Any attractor effect is an outright error that a stereotype induced.
In the underdetermined family, the head is an occupation ("The nurse who [verb] the [occupation] ... "). Here the head's own stereotype is the inherited term and the attractor's stereotype is the leakage term.
Across both families we vary several factors. The attractor can sit inside a relative clause or a prepositional phrase. There can be zero to three attractors. The attractor's tree distance to the prediction site can vary while its linear distance is held fixed, and vice versa. For each frame $s$ the leakage score is
$$L(s) = \log\frac{P(\text{fem} \mid s, a_F)}{P(\text{masc} \mid s, a_F)} - \log\frac{P(\text{fem} \mid s, a_M)}{P(\text{masc} \mid s, a_M)}.$$
Here $P(\text{fem}\mid\cdot)$ and $P(\text{masc}\mid\cdot)$ are the model's probabilities for the feminine and masculine completion (for example herself and himself), and $a_F$ and $a_M$ are attractor occupations stereotyped female and male. The inherited score $I(s)$ is the same quantity with the head occupation swapped instead of the attractor. A model with no off-structure leakage has $L = 0$.
To reach biases that gender swapping cannot handle, we build a parallel family for nationality and age. Examples are "The engineer from [country] who met the doctor from [country] was very ___." We score the sentiment of the model's top completions with a fixed lexicon, following the halo-effect dimensions of Investigating Subtler Biases in LLMs. The country pairs come from the gaps documented by the LLM Stereotype Index study.
Research Program
The first stage is purely diagnostic. We apply the probe to existing pretrained models, including GPT-2 and the open models evaluated in FairBelief. FairBelief found harmful associations at the top-1 prediction, which justifies scoring the top token rather than tail mass. The questions are whether $L$ is nonzero, whether it grows with the number of attractors, and whether it tracks linear distance more than tree distance.
The second stage is controlled training. We train matched GPT-2-small-sized decoders from scratch on one fixed English corpus that has been dependency-parsed once with a single off-the-shelf parser. The arms are a plain left-to-right objective, the DMLM objective, and a labeled-relation DMLM variant. We also fine-tune pretrained GPT-2 with the DMLM objective, reproducing the DM-GPT-2 setting, to test whether a structural objective added late can repair leakage that is already learned. Seeds, tokens seen and the hyperparameter search budget are identical across arms.
The third stage is mechanistic. We attribute the leakage term to attention from the prediction site to the attractor span, and we ask whether DMLM's mixture weights route around that span. We also study how leakage builds up over training, to test the feature-asymmetry prediction that weak contextual cues are overweighted early.
Baselines
The naive reference point is the plain left-to-right Transformer trained on the identical corpus, budget and seeds. Every claim is stated as a difference from it, in leakage $L$, inherited term $I$, and perplexity.
The strongest existing structural method is DMLM from Dependency-based Mixture Language Models, run as published but on our corpus. Its reported benefit is over 20 perplexity points on Penn Treebank and a UNION score of 85.31 versus 84.41 for GPT-2 on ROCStories. No reported result was found on Lacuna for any bias measure on DMLM. Beating the plain model would mean a reduction in mean $|L|$ outside seed variance, together with a flatter slope of $L$ against attractor count, at no worse perplexity.
The strongest existing debiasing competitor is counterfactual data augmentation, meaning gender-swapping the pretraining corpus. In Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods, augmentation plus debiased embeddings cut the WinoBias Type 1 pro- versus anti-stereotypical gap from 27.2 F1 (74.9 versus 47.7) to 2.1. We apply swapping to the same corpus with the same budget, both alone and combined with DMLM.
For general bias we report StereoSet, where GPT2-large reaches a language-modeling score of 88.3, stereotype score of 60.1 and idealized score of 73.0. This anchors our models' trade-off against the published lms–ss correlation.
Two ablations isolate the key ingredient. The first is DMLM trained on random trees that preserve each sentence's arc-length distribution. The second is a linear "predict the token $k$ ahead" auxiliary loss matched to real arc lengths. If either matches real-tree DMLM on leakage, the benefit comes from extra auxiliary supervision, not syntax.
Why not just…?
Why not just gender-swap the training data? This is the real threat for binary gender. Swapping makes occupations uninformative about gender, so it should shrink both $I$ and $L$, and the published 27.2-to-2.1 reduction shows how strong it is. We do not claim to beat it on gender. Instead the experiments test two things swapping cannot address. One is nationality and age leakage, where no swap dictionary exists and the subtler biases documented in the file persist. The other is whether swapping removes leakage from attractors that carry no swappable gender word. If swapping alone erases leakage on every attribute family, the structural contribution reduces to a measurement result, and we will report it that way.
Why not just apply corpus-level constraints at inference, as RBA does? Men Also Like Shopping cut amplification by 40.5% on imSitu with a 0.1% accuracy cost. It matches predicted marginals to training marginals across a whole batch, however. That cannot separate on-structure from off-structure influence, and it is not usable for single-prompt token-by-token generation. We include an RBA-style correction of the probe outputs as an upper-bound reference, to show it cannot drive $L$ to zero without also distorting $I$.
Why not just scale up? Large Transformers may already track structure well enough that leakage is negligible. Stage one settles this directly on pretrained models of increasing size. If leakage vanishes with scale, the training-stage experiments are dropped. FairBelief's finding that scale does not remove top-1 harmful beliefs (Vicuna-7B HONEST score 0.257) suggests this is unlikely, but it is an empirical question we test first.
Why not just control for sentence length? The parsing re-analysis shows that length explains apparent demographic effects. Our frames hold token length fixed and vary linear distance and tree distance independently. A length-only explanation therefore predicts no difference between relative-clause and prepositional-phrase attractors of equal length, which is testable.
Decisive Experiments
The first decisive check is whether, in the grammatically determined family, pretrained models show $L \neq 0$ at all. A second, separate check is whether leakage grows with attractor count. If $L$ is indistinguishable from zero, the line of work stops there.
The second check is the dissociation. Real-tree DMLM should reduce $|L|$ relative to the plain model, the random-tree and linear look-ahead ablations should not, and $I$ may stay flat or rise. That pattern would confirm that structure targets specifically the illegitimate component.
The third check is transfer to standard benchmarks. Lower probe leakage should coincide with a smaller pro- versus anti-stereotype gap on WinoBias Type 2 sentences, scored by language-model likelihood of the candidate antecedents. It should also show up on the nationality and age families. Winogender (Gender Bias in Coreference Resolution) and HONEST serve as secondary checks that general harm does not rise.
Throughout, the relative-clause versus prepositional-phrase asymmetry from Linzen and Leonard is the signature to watch. A model that leaks more across relative clauses is relying on linear shortcuts. A model that leaks more across prepositional phrases is processing structure the way humans do.
References
- StereoSet: Measuring stereotypical bias in pretrained language models
- Distinct patterns of syntactic agreement errors in recurrent networks and humans
- Feature-Wise Bias Amplification
- Dependency-based Mixture Language Models
- A Structured Language Model
- Dependency Language Models for Sentence Completion
- Is there Gender Bias in Dependency Parsing? Revisiting “Women’s Syntactic Resilience”
- Directional Bias Amplification
- Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
- Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models
- Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach
- FairBelief - Assessing Harmful Beliefs in Language Models
- Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
- Gender Bias in Coreference Resolution