Where Template Sensitivity Enters: Locating the Source of Unstable Bias Scores Inside Language Models

Social bias benchmarks for language models are fragile. Rewording a verb, adding an adjective, or changing the task instruction can halve a model's bias score or swap the order of two models on a leaderboard. The usual response is to report averages over many prompt variants. That treats the instability as noise without asking where it comes from. We propose to trace it inside the model. We run the same bias question under two templates that mean the same thing, copy internal activations from one run into the other layer by layer and position by position, and measure how much of the score gap each transplanted piece carries. This tells us whether the template changes the model's internal link between a demographic group and a stereotype, or only changes how that link is turned into an answer, for example whether the model picks "unknown."

The distinction matters in practice. If the association is stable internally and the instability sits in the last step that produces the answer, then benchmark fragility says little about the models' real biases, and a bias measure read from the stable internal site should rank models consistently across templates. If the association itself moves with wording, then no amount of prompt averaging will give a single number that means "this model's bias," and benchmarks should report bias as a function of template.

Core Claim

We hypothesize that most template-induced swings in bias scores for instruction-tuned decoder models are introduced late. They arise at the answer-option and final-token positions, where the model decides between committing to a person and abstaining. The mid-layer representation at the demographic mention tokens changes much less. We will test this hypothesis causally. If it holds, we will turn the stable site into a bias measure and check two things: that it is more reproducible across templates than behavioral scores, and that it still responds to real changes in bias. We treat the second check as seriously as the first, because a measure that never moves is perfectly stable and worthless.

The closest prior work motivates this plan but stops short of it. Social Bias Evaluation for Large Language Models Requires Prompt Variations varies instructions, few-shot examples and debiasing prompts for 26 models on BBQ, a multiple-choice bias benchmark for question answering. It shows ranking shifts with Kendall's $\tau$ and a max-minus-min "sensitivity gap." It treats the models as black boxes. The Tail Wagging the Dog applies meaning-preserving edits to WinoGender and BiasNLI. In one case, negating verbs dropped an ELMo model's BiasNLI bias score from 41.64 to 13.40 and inverted its ranking against RoBERTa. That work also uses no internal analysis, and it covers only small encoder and sequence-to-sequence models. On the internal side, Knowing Bias, Doing Better localizes "know-bias" feed-forward neurons in Llama, Qwen and Gemma using 45 fixed probing questions. It never asks whether those neurons behave the same way when the evaluation wording changes. We take the causal patching method from Interpretability in the Wild, which found a 28-head circuit that is 94% faithful to GPT-2 Small's behavior. Its advice to patch activations from real inputs rather than zero them constrains our design. Analysis Methods in Neural Language Processing supports the same constraint, because zeroing pushes hidden states into regions the model never sees.

Research Program

The basic unit is a template pair $(A, B)$: two meaning-preserving renderings of the same bias item. We draw pairs from three sources. The first is the instruction and few-shot variations of the prompt-variations study, applied to BBQ. The second is the synonym, descriptor, length and negation edits of Tail Wagging, applied to WinoGender and BiasNLI. The third is the explicit-demographic-marker probes of Template-Based Probes Are Imperfect Lenses, where writing "White doctor" produces artifacts because default groups are rarely named in text.

For each model we compute an aggregate bias score $b(\cdot)$ over the item set under each template. We then compute a recovery fraction for each internal site $s$. A site is a layer combined with an aligned token position: the demographic mention, the answer options, or the final token. Positions are aligned by role rather than by index, because edits change sentence length. The recovery fraction is

$$R_s = \frac{b(B \leftarrow A_s) - b(B)}{b(A) - b(B)},$$

where $b(B \leftarrow A_s)$ is the score when the model runs on template $B$ but its activation at site $s$ is replaced by the one from template $A$. $R_s$ near 1 means site $s$ carries the template effect, and $R_s$ near 0 means it does not. We compute this only for pairs where $|b(A)-b(B)|$ exceeds the item-bootstrap confidence interval, since otherwise the ratio is noise.

We split the outcome in two. BBQ's ambiguous-context score mixes how often the model abandons "unknown" with which person it picks when it does. We therefore compute $R_s$ separately for the abstention rate and for the direction of committed answers. This separates format sensitivity from stereotype direction before any internal claim is made.

From the recovery maps we pick the site whose representation of the demographic mention is most template-invariant. Invariance is judged by the cross-template similarity of a stereotype direction, estimated there by a difference of means between stereotyped and anti-stereotyped completions. From that site we define an internal bias measure: the projection of held-out items onto the direction. The key requirement is that the site is chosen by causal relevance and not only by probe accuracy. The site must also carry nonzero recovery for the committed-direction outcome on at least some template pairs. Otherwise it encodes demographic information the model does not use.

Baselines

The naive reference point is the standard single-template behavioral score. This means the default BBQ prompt, the original WinoGender and BiasNLI templates, and StereoSet's stereotype score. Its cross-template rank agreement is what everything else must improve on. Tail Wagging's 41.64-to-13.40 swing and ranking inversion show how poor this can be.

The strongest behavioral competitor is prompt-variation averaging as in Social Bias Evaluation for Large Language Models Requires Prompt Variations. Each model's bias is the mean over $K$ templates. We run it with the same template pool, and we match forward passes: patching costs roughly one extra forward pass per site per item, so averaging gets $K$ equal to our total pass budget. That study reports ranking fluctuations through Kendall's $\tau$ and sensitivity gaps, but the file quotes no scalar value. There is no reported result found on Lacuna for the split-half ranking stability of prompt-averaged BBQ scores, so we measure it ourselves. The internal measure beats averaging only if its split-half Kendall's $\tau$ across disjoint template sets is higher at equal compute, with confidence intervals from bootstrapping over models and items that do not overlap.

The strongest internal competitor is KnowBias attribution from Knowing Bias, Doing Better. Its Integrated-Gradients neuron sets are recomputed under each template, and bias is measured by their mean activation. This tests whether an existing localization is already template-stable. We report the overlap of the attributed neuron sets across templates alongside our site-level invariance. The file gives KnowBias an average rank of 2.1 across BBQ, CrowS-Pairs and StereoSet as a mitigation method. That figure does not bear on measurement stability, so it is context only.

The ablation that isolates our key ingredient replaces causally selected sites with the layer and position where a plain linear probe decodes the demographic best. If that probe-selected measure is just as stable and valid, the patching step is unnecessary. We also include Prompt Bias Calibration from Removing Prompt-template Bias in Reinforcement Learning from Human Feedback, adapted so a linear head predicts the template-only score shift from the final prompt token and subtracts it. All methods are tuned on the same development templates and evaluated on held-out template families.

Why not just…?

Why not just average over many paraphrased prompts? This is the real threat. With enough templates, the averaged score may become stable. The prompt-variations study found that ranking swaps persisted under its variations, and Tail Wagging found that subsampled occupation lists gave overlapping score distributions across different models. Neither tested averaging at large $K$ against a held-out template family, so the question is open. Our compute-matched split-half comparison settles it. If averaging matches the internal measure, we will conclude that the internal measure's value is diagnostic rather than practical, and we will say so.

Why not just run a factorial analysis of variance over prompt components? Crossing instruction, demonstration, marker and wording factors, as in the component ablations of The Death and Life of Great Prompts, would tell us which input part moves the score. It is cheap, and we run it as a first step to choose which template pairs to patch. It cannot tell whether the moved score reflects a changed association or a changed answer format, and that is the question that decides whether benchmark instability is about bias at all.

Why not just split BBQ's score into abstention and direction and stop there? If the whole swing lives in the abstention rate, the output-level split already supports the late-readout hypothesis, and internal analysis adds little for BBQ. We report this split first. Patching is needed for the committed-direction component and for WinoGender, BiasNLI and the marker probes, which have no abstention option.

Why not just abandon templates and probe with natural text? Comparing Template-based and Template-free Language Model Probing shows that natural text changes probe outcomes by up to 42% Acc@1 and brings in uncontrolled confounds. The markedness findings imply that natural text rarely supplies balanced counterfactual pairs. Patching requires paired inputs, so we stay with templates and treat their variation as the experimental manipulation.

Proposed Experiments

We will study open decoder models at two or three scales from the Llama, Qwen and Gemma families used by KnowBias. We add BERT with and without MABEL (MABEL, via princeton-nlp/MABEL) to connect with the encoder-era findings of Measuring Fairness with Biased Rulers. That paper found that the [CLS] token absorbs template context and that bias scores correlate near zero or negatively across templates. Our site-level analysis of [CLS] versus target-token positions directly tests that mechanism.

The data are BBQ (BBQ) under the prompt-variation families; WinoGender and BiasNLI under the Tail Wagging edits; the Regard and NS-Prompts marker probes; and StereoSet and CrowS-Pairs as secondary checks. The first result is the recovery maps, reported per model and per perturbation type, for both the abstention and the direction outcome.

There are three decisive checks. The first is localization. The hypothesis is supported if recovery for the committed-direction outcome concentrates at final-token and option positions in late layers. It is refuted if demographic-mention positions in middle layers carry most of the recovery. We expect the marker probes to be the likeliest exception, since the markedness artifact may alter the association itself. We will report such exceptions rather than average them away.

The second is stability. We compare the internal measure's split-half Kendall's $\tau$ across held-out template families against single-template scores, compute-matched averaging, KnowBias activations, the probe-selected ablation and calibration.

The third is validity. The internal measure must move in the right direction under controlled bias changes: KnowBias neuron scaling at several strengths, and MABEL versus base BERT. For occupation and gender items, it must track the US census labor statistics that Trustworthy Social Bias Measurement used, where their DivDist measure reached a Spearman correlation of 0.83. That paper's "debiasing mirage" gives a sharper test. If debiasing lowers behavioral scores but leaves the internal measure unchanged at a causally relevant site, that is evidence the intervention hides bias from certain templates rather than removing it.

A measure that is stable but fails the validity checks counts as a negative result. So does a localization that shows the association itself shifting with wording. Either outcome would still change how bias leaderboards should be reported.