When interpretability researchers look inside a trained language model for the neurons that carry a shortcut, they compute a score for every neuron in that one model and keep the top few. Comparative genomics faced the same problem decades ago and found a way around it. A transcription-factor binding motif six to twelve DNA letters long turns up millions of times in a genome by chance. Biologists separate real regulatory sites from chance matches by keeping only those preserved across related species, because natural selection protects functional sites while random sequence drifts. We propose the analogous filter for neural networks. We train several models that differ only in random seed, align their neurons into a shared coordinate system, and declare a neuron "bias-mediating" only if its aligned counterparts mediate the bias in the other models too. We then remove only these conserved neurons.
The claim is that single-model attribution mixes two kinds of hits. Some reflect how the task and data push any network toward a shortcut. Others are idiosyncratic to one optimization run. Only the first kind is worth editing. Cutting the second kind costs accuracy and buys no robustness. If the claim holds, a conservation-filtered mask should beat a same-sized single-model mask on out-of-distribution challenge sets at a lower in-distribution cost. It should also make a prediction single-model methods cannot make: a mask derived from seeds A, B and C, mapped through the alignment, should work almost as well on a held-out seed D as D's own mask.
Why this gap is real
The motivating evidence comes from both sides. In interpretability, Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? shows that causal criteria within one network admit huge numbers of incompatible "perfect" circuits. In toy multilayer networks, the median count rises from 38 to 910,000 as width grows from 2 to 5 neurons. That paper suggests judging explanations by their usefulness. It does not propose comparing circuits across independently trained models.
The bias-localization work all operates on a single checkpoint. Investigating Gender Bias in Language Models Using Causal Mediation Analysis measures each component's "indirect effect": how much of a bias flows through that component when its activation is swapped between a stereotyped input and a counterfactual one. It finds that 10 of 144 attention heads in GPT-2 small carry almost the whole indirect effect. Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis uses the same neuron-level indirect effects in BERT-base to mask shortcut neurons and then unlearn them. Debiasing Masks learns sparse pruning masks for the same purpose.
These papers also disagree about where shortcuts live. Debiasing Masks reports early query/key heads, while the mediation-analysis work on BERT reports layers 10 and 11. Seed-specific noise in attribution is one plausible explanation for such disagreement.
Genomics supplies the template and also its main warning. Of mice and men: phylogenetic footprinting aids the discovery of regulatory elements. reports that human–mouse filtering in ConSite removed up to 85% of false-positive binding-site calls. The same paper states the "Goldilocks" problem: species that are too close share everything, so the filter does nothing, and species that are too far apart share nothing. c-REDUCE adds the part we reuse most directly: it combines conservation with a functional signal (expression) rather than using conservation alone. It recovered 18 versus 10 motifs on hard yeast transcription factors.
For alignment we borrow the method of Git Re-Basin, which matches hidden units across independently trained networks by solving assignment problems so that the networks can be merged. Git Re-Basin aligns models in order to average their weights. We use alignment instead as a coordinate system for asking whether a causal role recurs across models.
Core Claim
Let $K$ independently trained models be aligned to a reference model $r$. The alignment $\pi_k$ maps neuron $i$ of $r$ to its matched neuron in model $k$. Let $\mathrm{NIE}_k(j)$ be neuron $j$'s indirect effect on the shortcut in model $k$, and $\mathrm{NIE}^{\mathrm{ctl}}_k(j)$ its indirect effect under a control intervention that changes the label but not the shortcut cue. We define the conservation score
$$C_i = \frac{1}{K}\sum_{k=1}^{K} \mathbb{1}\left[\pi_k(i) \in \mathrm{Top}_q(\mathrm{NIE}_k - \mathrm{NIE}^{\mathrm{ctl}}_k)\right],$$
where $\mathrm{Top}_q$ is the top $q$ fraction of neurons by bias-specific effect. In words, $C_i$ is the fraction of models in which the aligned counterpart of neuron $i$ ranks among the most bias-specific mediators.
The control term plays the role of c-REDUCE's functional signal. It excludes neurons that are conserved and influential in general, such as hub neurons and general linguistic machinery. Those are the analogue of genes under purifying selection and should be left alone.
We predict three things for masks matched on neuron count:
- Masks built from high-$C$ neurons give larger challenge-set gains per neuron removed than single-model top-$k$ masks.
- They lose less accuracy in distribution.
- They transfer to held-out seeds.
Conservation does not resolve non-identifiability in principle. Many models could share the same degenerate circuit. It does supply an external selection criterion, and its value can be tested by the usefulness standard that Méloux et al. propose.
Research Program
The main testbed is natural language inference (deciding whether one sentence follows from another) and paraphrase detection. These are the settings where mediation-based masking already has published numbers.
We will build three nested "evolutionary distances." The closest is fine-tuning one pretrained BERT-base from $K$ seeds. Here neuron coordinates are inherited, so no alignment is needed, but conservation may simply reflect shared pretraining; this is the human–chimp regime. The middle distance is fine-tuning from BERT checkpoints pretrained with different seeds. If no such checkpoints are available, we pretrain smaller BERT-style encoders from several seeds under a fixed budget. This regime needs permutation alignment by activation matching on a held-out corpus. The farthest distance aligns across model widths. We expect it to degrade, and if it does, that tells us where the useful range ends.
For each model we compute neuron-level indirect effects for the lexical-overlap shortcut, following Sae Lim et al. We compute the conservation score above and then apply their masking-plus-gradient-unlearning procedure, changing only which neurons are selected.
An analysis track asks a second question with the same machinery. Do the layer depths of conserved neurons settle the early-versus-late disagreement between Debiasing Masks and the BERT mediation results, or are the published depth profiles mostly seed-specific?
A smaller extension tests whether the effect carries over to social bias. We will repeat the procedure for gender bias on WinoBias, following Vig et al., using small GPT-2-style models pretrained from several seeds. Here the readouts are total and indirect effect, plus perplexity as the utility check that guards against the "bad language model trap" of An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models. That survey shows debiasing that doubles GPT-2 perplexity from 30.15 to 65.49 while appearing to improve bias scores.
Baselines
The naive reference point is the unedited fine-tuned model, averaged over all $K$ seeds. Debiasing Masks reports 61.97 on MNLI→HANS for its baseline. No intervention should be judged against a single lucky seed.
The strongest existing method is the causal-grounded masking with gradient unlearning of Sae Lim et al. It reports HANS accuracy rising from 66.79% to 70.05%, PAWS macro-F1 from 43.04% to 54.42%, and an in-distribution cost of typically 1–2%. We will rerun it ourselves on every seed with identical unlearning hyperparameters and an identical masked-neuron budget, so the only difference is the selection rule. To be fair on compute, the single-model version gets $K$ times as much attribution data as each model contributes to our pooled score. Beating it means a higher mean HANS accuracy and PAWS F1 across seeds at equal or lower MNLI/QQP accuracy loss, with the gap exceeding seed-to-seed standard deviation.
The second strongest is Debiasing Masks, which reports 68.69 on HANS and 44.34 on PAWS. We will run it on the same seeds with the same sparsity. On FEVER Symmetric we found no reported result on Lacuna for either method, so we will report our own reruns only.
The key ablation removes the alignment step in the different-pretraining regime. Neurons are matched by raw index, which is meaningless across separately pretrained models. If the gains survive this, conservation is not what produced them. A second ablation drops the control term, to check whether bias-specificity matters or whether conserved high-effect neurons suffice. A third gives the single-model method the same mask size as the high-$C$ set, so that sparsity alone cannot explain the gains.
Why not just…?
Why not just resample the probe data within one model and keep the stable neurons? This is the most serious alternative. Bootstrapping attribution over subsets of the counterfactual probes removes measurement noise at a fraction of the cost. It cannot remove optimization noise, because a neuron that idiosyncratically carries the shortcut in one model will be stable across resamples of that model. We will run bootstrap-stability selection as a full baseline. If it matches conservation filtering on challenge sets and on held-out-seed transfer, the cross-seed machinery is unnecessary and we will say so. The transfer test is where we expect them to differ.
Why not just ensemble or average the seeds? Averaging predictions, or merging weights through Git Re-Basin, reduces variance. Every seed has learned the lexical-overlap shortcut, so the ensemble keeps it too. We will report the ensemble's HANS and PAWS scores. We will also run single-model mediation analysis on a Git Re-Basin-merged model, which tests whether merging before attribution achieves the same filtering more cheaply.
Why not just tune the number of masked neurons in a single model? Lin et al. in Bi-directional Bias Attribution already need grid searches over how many neurons to silence. A better-tuned top-$k$ may close the gap. Our matched-budget comparison sweeps $k$ for the single-model method with the same grid we give our method. Conservation must win along the whole accuracy-versus-robustness curve, not at one point.
Why not just fine-tune on counterfactual or reweighted data? Global retraining needs no localization. The mitigation literature reports forgetting and perplexity damage from it, and Debiasing Masks reports localized masking above standard debiasing (68.69 vs 67.06 on HANS). We include standard debiasing as a reference, but the question here is about selecting components, not about whether editing beats retraining.
Decisive Checks
The proposal stands or falls on three results across seeds:
- Matched-budget comparison. On the different-pretraining-seed models, conservation-filtered masks must beat single-model and bootstrap-stability masks of equal size on HANS and PAWS, at equal or smaller MNLI/QQP cost.
- Held-out-seed transfer. A mask computed on $K-1$ seeds and mapped to the held-out seed must recover most of that seed's own-mask gain. A raw-index mapping must fail.
- Distance profile. Filtering power must be weakest in the shared-pretraining regime and strongest at intermediate distance, as the Goldilocks argument predicts.
If conserved and single-model masks perform identically, then seed-specific attribution noise is not what limits bias localization, and the identifiability concern matters less in practice than Méloux et al. suggest. That is still a useful negative result.
References
- Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?
- Investigating Gender Bias in Language Models Using Causal Mediation Analysis
- Identifying and Mitigating Annotation Bias in Natural Language Understanding using Causal Mediation Analysis
- Debiasing Masks: A New Framework for Shortcut Mitigation in NLU
- Of mice and men: phylogenetic footprinting aids the discovery of regulatory elements.
- c-REDUCE: Incorporating sequence conservation to detect motifs that correlate with expression
- Git Re-Basin: Merging Models modulo Permutation Symmetries
- An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models
- Bi-directional Bias Attribution: Debiasing Large Language Models without Modifying Prompts