Reversal Probes for Detecting Memorized Fairness in Bias Benchmarks

Social bias benchmarks such as StereoSet and BBQ are public, years old, and widely mirrored on the web. Any large language model trained on web crawls may therefore have seen their exact items, and possibly their answer keys. When such a model scores as "fair," we cannot tell whether it is actually neutral or whether it is recalling memorized text. This proposal starts from one observation. A memorized benchmark item is memorized in one direction only: the order in which the item poses its question. Bias measurement research has shown that a model's associations can be probed in two directions. One direction asks whether a demographic attribute pushes the model toward a task answer. The other asks whether a task cue pushes the model toward inferring an attribute. If we rewrite each benchmark item so it asks the reverse question, the rewrite should track the model's real associations, while the canonical item tracks what the model memorized. The gap between the two directions, measured against the gap expected for items the model has never seen, can then serve as an item-level contamination signal that is specific to fairness evaluation.

What would be new is a measurement claim, not a debiasing method. On fairness benchmarks, contamination shows up as excess directional asymmetry. That asymmetry can be measured from the existing items alone, without logits from training data and without writing a new benchmark. We also hypothesize, and will test rather than assume, that this signal survives the clean reinforcement-learning phase that is known to erase loss-based contamination signatures.

Background and the gap

The directional view of bias comes from Directional Bias Amplification. That paper splits bias into an attribute-to-task pathway (A→T) and a task-to-attribute pathway (T→A), and shows that the two can move in opposite directions. Masking people in images removed A→T bias but increased T→A bias. DPA replaces co-occurrence counts with a trained probe that measures how predictable the attribute is, and ML-EAT shows that aggregate association scores often hide one-sided shifts. These works supply our measurement lens. They also impose a constraint we must respect: real models have genuine directional asymmetries, so a raw asymmetry cannot by itself indicate contamination.

On the contamination side, Generalization or Memorization detects leaks by noticing that sampled outputs on contaminated items collapse onto one string. ABCD: All Biases Come Disguised shows that models have lower perplexity on the canonical option order of MMLU-Pro, and that letter-position heuristics can carry models to above 95% on nonsense questions. On The Fragility of Benchmark Contamination Detection in Reasoning Models shows that one clean round of GRPO pushes loss-based and membership detectors toward chance. All of this work targets math, code, and knowledge benchmarks.

Bi-directional Bias Attribution is the closest work that touches both directions. It traces forward and backward bias neurons in Llama-3.1, Llama-3.2, and Mistral, and it reports StereoSet religion SS falling from 56.94% to 51.35%. It never asks whether those test items were in pretraining. Its use of two directions is a debiasing tool, not a contamination diagnostic. That untested assumption motivates this proposal.

Core Claim

Every benchmark item has a canonical direction. A StereoSet intrasentence item gives a group and asks the model to prefer a stereotypical or anti-stereotypical attribute, which is A→T. A BBQ item gives a context and a question about a trait and asks which person it describes, which is T→A. For each item $i$, we build a reversal probe that keeps the same group, trait, and context but queries the opposite conditional. For StereoSet, the probe gives the attribute sentence and asks which group fills the slot. For BBQ, it names the person and asks a yes/no question about the trait.

Let $b_c(i)$ and $b_r(i)$ be the normalized bias scores in the canonical and reversed directions, both on a scale where 0 means neutral. Let $\delta_k$ be the natural asymmetry for bias category $k$. We estimate $\delta_k$ on freshly written templates that the model cannot have seen. The contamination score is

$$g(i) = \big(b_c(i) - b_r(i)\big) - \delta_{k(i)},$$

that is, the excess of the canonical-minus-reversed gap over what clean items in the same category show. If an item's answer key was memorized, $b_c$ moves toward the memorized label, whether stereotypical or "fairness-washed," while $b_r$ stays near the model's genuine association, so $|g|$ grows. The sign of $g$ is informative: it says whether contamination made the model look fairer or more biased than it actually is.

Research Program

The work has three parts.

The first part is construction. For StereoSet and BBQ, we write deterministic reversal probes for every item. We also hand-write a fresh-template set in each bias category, measured in both directions, to estimate $\delta_k$ and to serve as ground truth for the model's genuine bias. To keep BBQ's multiple-choice format from confounding everything, all BBQ scoring uses the Matched-and-Dashed protocol from ABCD. That protocol presents unlettered options and matches the model's full-text answer to an option.

The second part is controlled contamination, which gives known ground truth. We split each benchmark into an injected half and a held-out half and fine-tune open models under four conditions. Following the model families used by A Narrowing Geometry in Contaminated Reasoning, we use Qwen2.5-1.5B and 7B, plus Llama-3.2-3B. The first condition is label-level contamination with the original answers. The second is label-level contamination with only anti-stereotypical or "Unknown" answers, which mimics a developer polishing a fairness leaderboard. The third is data-level contamination with inputs only, using the levels defined in the contamination survey. The fourth repeats each of these conditions followed by one epoch of clean GRPO, as in the fragility study.

The third part is a natural audit. We search FineWeb by n-gram match for verbatim BBQ and StereoSet templates. We then compute $g$ on off-the-shelf Llama-3.1-8B and Mistral-7B, and test whether high-$g$ items are enriched among the web-matched ones.

Baselines

The naive reference point is to report canonical StereoSet SS and BBQ bias scores as published. Any method that does not change the conclusion relative to this reference adds nothing.

The strongest competitor at the level of a whole benchmark score is fresh-template rewriting: newly written items in the same categories. We use this as ground truth, not as a rival. A method beats the naive reference if its contamination-corrected score lands closer to the fresh-template score. Fresh templates cannot say which existing items are compromised, so they do not compete on item-level detection.

For item-level detection, the strongest existing detectors are Min-K%, loss thresholding, and LiRA. In the fragility study, LiRA reached 89.1% AUROC after supervised contamination, then lost about 14 points after one GRPO epoch, and all ten detectors fell to 50–58% AUROC for contaminated reasoning models. Those figures come from math benchmarks. There is no reported result on Lacuna for contamination detection on BBQ or StereoSet. We also run CDD from Generalization or Memorization, which improved detection by 21.8–30.2% over n-gram and LLM baselines on DETCON, and the canonical-versus-permuted option perplexity test from ABCD.

All detectors see the same models, the same item splits, and the same budget of forward passes per item. For CDD that budget means the same number of samples. Any thresholds are tuned on one shared validation split. Our method wins if it has higher AUROC than every baseline after GRPO. Before GRPO, matching LiRA is enough.

Two ablations isolate the key ingredient. The first drops the calibration term $\delta_k$ and uses the raw asymmetry, which tests whether genuine directional bias would otherwise produce false alarms. The second replaces the reversal probe with a paraphrase that keeps the same direction, which tests whether reversing the conditional matters or whether any surface rewrite works.

Why not just…?

Why not just write fresh templates and compare scores? This is the most serious alternative. If the only goal is a corrected score for the whole benchmark, it may be enough, and we report it as ground truth for that reason. It does not identify which published items, or which bias categories, are compromised. It also needs new human-written items for every audit. If $g$ adds nothing beyond the fresh-template estimate of $\delta_k$, the item-level detection AUROC will show it.

Why not just run Min-K%, LiRA, or CDD on BBQ? The fragility study shows that loss-based signals collapse toward chance after clean RL, and production chat models are post-trained this way. We hypothesize that a gap between directions is harder for RL to erase, because RL compresses likelihood gaps on the trained direction rather than aligning both directions. This is an untested bet. The GRPO conditions decide it directly. If $g$ also collapses after GRPO, we will report that as a negative result.

Why not just paraphrase the benchmark? Dong et al. report that paraphrased items in HumanEval_R and the DETCON implicit split still drew the same collapsed output distribution. Paraphrase keeps the conditional the model was trained on. The same-direction paraphrase ablation tests whether reversal matters beyond rewording.

Why not just fix the multiple-choice format? The Matched-and-Dashed protocol removes letter and position artifacts. It does nothing about memorized answer content. We adopt it as a control, so that any remaining $g$ cannot be explained by option order.

Decisive Checks

Three results would settle the claim. First, in the controlled setting, $g$ must separate injected from held-out items, with AUROC well above chance both before and after GRPO, and the calibration ablation must show that $\delta_k$ reduces false positives on clean models. Second, in the "fairness-washed" condition, canonical SS should approach 50% while reversed probes and fresh templates stay biased, and the sign of $g$ should correctly flag that the model was made to look fairer. Third, on off-the-shelf models, high-$g$ items should be over-represented among items with verbatim matches in FineWeb. If off-the-shelf models show no excess asymmetry, published fairness scores on these benchmarks gain a measure of trust they currently lack. That outcome is also worth reporting.