Interpretability researchers increasingly find "circuits" by machine: an algorithm switches off connections inside a language model one at a time, measures how much the model's output changes on a set of test prompts, and keeps the connections whose removal hurts most. These searches routinely screen tens of thousands of candidate connections against a few hundred prompts, often generated from a handful of sentence templates. Neuroimaging has a name for what happens when you screen tens of thousands of variables on a small sample and report the ones that pass a threshold. It is the winner's curse: the effects that survive are mostly the ones that noise happened to inflate, and they shrink or vanish when someone tries to replicate them. This proposal asks whether automated circuit discovery has the same disease. If it does, the proposal provides a cheap statistical correction and a rule for how many prompts, and how many distinct templates, a circuit claim needs.
The key observation is that the relevant sample size is not the number of prompts. Prompts generated from the same template behave like repeated trials from the same brain-scan participant. They are strongly correlated, so a 500-prompt set built from 15 templates may carry the statistical weight of roughly 15 observations. That is the same regime as the $n=20$–$50$ cohorts that produced unreplicable connectome findings. We propose to measure how badly edge effects are inflated at realistic prompt budgets. We then build a discovery procedure that separates selection from estimation and resamples by template rather than by prompt, and we test whether its circuits replicate on new template families where current circuits do not.
Core Claim
Edges selected by ACDC or by Edge Attribution Patching with Integrated Gradients (EAP-IG) at typical prompt budgets have systematically inflated effect estimates. A substantial fraction of them fail to reappear when discovery is rerun on an independent draw of prompts. The failure is driven mainly by the number of templates, not the number of prompts. A procedure with three parts should restore replicability at little extra compute: discover on one set of templates, estimate effects on disjoint templates, and control the false discovery rate with template-clustered resampling. This is a claim about the reliability of the evidence behind circuit claims, not about any particular circuit.
Where this sits in prior work
Two papers supply the methods we audit. Towards Automated Circuit Discovery for Mechanistic Interpretability introduced ACDC, which prunes edges whose removal changes the output distribution by less than a fixed threshold. It recovered known circuits, isolating 68 of 32,000 edges on Greater-Than. MIB: A Mechanistic Interpretability Benchmark standardized circuit localization with integrated metrics, the circuit performance ratio (CPR) and the circuit–model distance (CMD), and found EAP-IG among the strongest methods. Neither paper asks whether the edge set itself is stable under resampling of the prompts. Both choose thresholds by heuristic, and neither corrects for testing thousands of edges at once.
The closest existing work motivates the gap rather than filling it. Transformer Circuit Evaluation Metrics Are Not Robust showed that faithfulness scores swing with ablation choices: node versus edge, mean versus resample, and token positions. On compiled Tracr models, the "true" circuit even inverted between zero and resample ablation. That paper varies how the intervention is performed while holding the prompt set fixed. We hold the intervention fixed and vary the sample, which is a separate source of instability it does not examine. Its result also constrains our design. We cannot treat Tracr ground truth as unconditional, so every analysis fixes one ablation protocol and reports results per protocol.
The statistical lens comes from the small-sample literature. Winner's Curse? On Pace, Progress, and Empirical Rigor argues that ML empirical practice rewards selected results without rigor checks. Beating the Winner's Curse via Inference-Aware Policy Optimization treats the inflation that follows selection-then-estimation as something an algorithm should account for. Neuroimaging work makes the same point concretely. Reduced White Matter Integrity With Cognitive Impairments in End Stage Renal Disease is a small-cohort study whose effects could not survive rigorous multiple-comparison correction. A functional connectome phenotyping dataset including cognitive state and personality measures documents how $n=20$–$50$ cohorts produced contradictory connectivity–phenotype associations. We borrow their diagnosis and their tools, namely held-out replication, false discovery rate control, and cluster-aware power analysis. We do not borrow their data.
We deliberately do not pursue the other obvious bridge, which is using circuits to break the "dimensionality bottleneck" that Scaling laws for language encoding models in fMRI hypothesized for the 30-billion-parameter plateau in brain encoding. A skeptic would rightly answer that principal components or stronger ridge penalties likely do the same job more cheaply. With only three subjects, that comparison would also be hard to settle.
Research Program
The design copies the subsampling studies that population biobanks made possible in neuroimaging. For each task and model, we first build a "population": a very large prompt pool drawn from many templates. For IOI, this means expanding the template families well beyond the standard set to several hundred templates with thousands of name and object fillers. Greater-Than gets the same treatment, and so do MIB's arithmetic and multiple-choice tasks. We run discovery on the full pool with a fixed ablation protocol and treat the resulting edge set and per-edge effects as the reference circuit. We confirm that this reference is itself stable by checking that two disjoint halves of the pool agree.
We then draw many independent "studies" from the pool. Each study crosses a template count $T$ from 2 to 100 with a prompt count $N$ from 50 to 5,000. We run ACDC and EAP-IG on each draw at their default thresholds and record which edges are selected and what effect each one is estimated to have.
Three quantities summarize the curse. The first is replication: the overlap, by Jaccard similarity, between edge sets from two independent studies of the same size, and the precision and recall of each against the reference circuit. The second is the inflation ratio for selected edges,
$$R = \frac{\bar{\delta}_{\text{disc}}}{\bar{\delta}_{\text{ref}}},$$
where $\bar{\delta}_{\text{disc}}$ is the mean effect of the selected edges as estimated on the prompts that selected them, and $\bar{\delta}_{\text{ref}}$ is the mean effect of those same edges on the reference pool. A value $R \gg 1$ is the winner's curse. The third is the effective sample size. If edge effects are correlated within templates, the standard design effect from cluster sampling gives
$$N_{\text{eff}} = \frac{N}{1 + (m-1)\rho},$$
where $m = N/T$ is prompts per template and $\rho$ is the intraclass correlation of an edge's effect across prompts from the same template. We estimate $\rho$ per edge and test whether replication curves collapse onto a single curve when plotted against $N_{\text{eff}}$ rather than $N$.
The corrective procedure has three parts. First, discovery and estimation happen on disjoint templates, not merely disjoint prompts. Second, each edge receives a p-value from a template-clustered permutation or bootstrap of its effect. Third, the selected set is controlled at a target false discovery rate with the Benjamini–Hochberg procedure. Cross-fitting swaps the roles of the two template halves so that no data are wasted. The output is a circuit whose edges carry calibrated uncertainty, along with a power curve that tells a researcher how many templates a given edge effect size requires.
The models are GPT-2 Small, Qwen-2.5 0.5B, and Gemma-2 2B from MIB, with Llama-3.1 8B as a stretch goal using EAP-IG only, since repeated ACDC runs at that scale are expensive. Compiled Tracr models give a setting with known structure, with the caveat above about ablation dependence.
Baselines
The naive reference point is the current practice. ACDC or EAP-IG is run once on the standard prompt set at its default threshold, and faithfulness is reported on prompts from the same templates. This is the condition our replication and inflation measurements are designed to indict. If it already replicates well at standard budgets, the paper's central claim fails.
The strongest existing method is EAP-IG as configured in MIB: A Mechanistic Interpretability Benchmark, which reports a CMD of 0.03 on IOI with GPT-2 Small. We run it on identical prompt draws, with its integration steps and threshold sweep tuned on the same budget we give our procedure. Beating it does not mean a lower CMD on the benchmark's own split. It means higher precision against the reference circuit and higher cross-study replication at matched circuit size. It also means CPR and CMD on held-out template families that stay closer to their in-template values.
The third baseline is stability selection. We bootstrap prompts (not templates), rerun discovery on each resample, and keep edges selected in most resamples, matching the total number of forward and backward passes our procedure uses. This is the most natural fix a practitioner would reach for, so it is the most serious competitor.
Two ablations isolate the key ingredients. The first is our procedure with prompt-level rather than template-level splitting and resampling. If this matches the full method, the template-clustering claim is wrong. The second is our procedure without false discovery rate control, using sample splitting alone, which separates the debiasing contribution from the multiple-testing contribution.
Why not just…?
Why not just use more prompts? They are free for language models. Prompts from a fixed template set are cheap, but our hypothesis is that extra prompts from the same templates add little information once $\rho$ is high. The design-effect analysis tests this directly. If replication keeps improving with $N$ at fixed $T$, this objection wins, and the recommendation reduces to "use bigger prompt sets." Templates, by contrast, are costly to write and validate. Many behaviors people want to explain, such as specific failure modes or rare capabilities, come with few natural examples.
Why not just bootstrap and keep stable edges? Stability selection is a real threat and is run as a baseline. Our concern is specific. Bootstrapping prompts within a fixed template set measures stability against prompt noise, but it cannot reveal edges that are stable only because every resample shares the same template artifacts. The decisive comparison is replication on held-out template families. If prompt-level stability selection matches our template-clustered procedure there, we will report that simpler method as the recommendation.
Why not just report faithfulness on held-out prompts? Held-out faithfulness checks whether the whole circuit still works. A circuit padded with noise-selected edges can still score well, because extra edges rarely hurt faithfulness, and the hydra effect described in Open Problems in Mechanistic Interpretability means backup components can compensate. The edge-level claims researchers actually make ("head 9.6 moves names") need edge-level replication, and an aggregate score does not provide it. We report both measures to show where they disagree.
Proposed Experiments
The first decisive check is whether the curse exists. On IOI and Greater-Than in GPT-2 Small, with the standard template budget, we estimate the inflation ratio $R$ and cross-study Jaccard overlap for both ACDC and EAP-IG. No reported result was found on Lacuna for edge-set replication or effect inflation under prompt resampling, so these numbers are new. A finding of $R$ near 1 and high overlap would end the project's main thread. We would still publish it as evidence that current circuit claims are statistically sound.
The second check is whether templates, not prompts, govern reliability. We fit replication against $N$ and against $N_{\text{eff}}$ across the full $T \times N$ grid. The template hypothesis predicts that curves at different $T$ collapse only under $N_{\text{eff}}$.
The third check is whether the correction works where it matters. We train every method on a subset of template families and evaluate on unseen families. The metrics are MIB's CPR and CMD, precision and recall against the reference circuit, and the realized false discovery proportion, which should sit near the target rate if the procedure is calibrated. Repeating this on MIB's arithmetic task is especially informative. MIB found that carry-like variables resist clean localization across models, and our framework predicts this shows up as low $\rho$-adjusted power rather than as a stable but strange circuit.
Finally, on Tracr, under each ablation protocol separately, we test whether false discovery rate control at a fixed level recovers the compiled structure. This connects our sample-size axis to the ablation-choice axis in Transformer Circuit Evaluation Metrics Are Not Robust, and it lets us report which of the two sources of instability dominates at realistic budgets.
References
- Towards Automated Circuit Discovery for Mechanistic Interpretability
- MIB: A Mechanistic Interpretability Benchmark
- Transformer Circuit Evaluation Metrics Are Not Robust
- Winner's Curse? On Pace, Progress, and Empirical Rigor
- Beating the Winner's Curse via Inference-Aware Policy Optimization
- Reduced White Matter Integrity With Cognitive Impairments in End Stage Renal Disease
- A functional connectome phenotyping dataset including cognitive state and personality measures
- Scaling laws for language encoding models in fMRI
- Open Problems in Mechanistic Interpretability