Interpretability researchers now routinely extract "circuits" from language models. A circuit is a tiny subnetwork, often 1–7% of the parameters, that alone reproduces a behavior such as resolving which name a sentence refers to. Every such circuit has been judged with the weights it inherited from the trained model. A separate result in network compression says that for structured pruning, the inherited weights barely matter: the pruned shape, retrained from random initialization, does as well or better (Rethinking the Value of Network Pruning). This proposal joins the two. We freeze the wiring of a discovered circuit, throw away its weights, retrain it from random initialization on the task, and ask whether it relearns the behavior. We also ask whether it relearns the same mechanism, meaning the same roles in the same heads.
Neuroscience supplies the key comparison. Monosynaptic rabies tracing labels only the direct inputs of chosen "starter" neurons. Its outputs are usually summarized as input-proportion profiles, meaning what fraction of a neuron's inputs comes from each brain region. Brain-wide presynaptic networks of functionally distinct cortical neurons found that neurons with different function share local wiring but differ in these long-range proportions. That raises a question biology cannot easily test: is such a profile enough to specify a computation, or does function need the exact edge-by-edge wiring? In a transformer we can build every level of description and train it. The contribution is a graded test of which level of wiring description is sufficient for a network to relearn a task: per-layer budget, per-node input profile, or exact edges.
Core Claim
We claim that circuits found by joint weight-and-edge pruning are more than readouts of trained weights. They also carry architectural inductive bias, a built-in structural preference that makes some solutions easier to learn. Concretely, we predict three things when the retained weights are reinitialized and retrained:
- Exact circuit wiring will relearn the target task with better held-out generalization and sample efficiency than random subgraphs matched in per-layer budget.
- A good part of that advantage should survive in random subgraphs that preserve each node's input-proportion profile.
- Retrained circuits will re-form recognizably similar head roles.
Each prediction could fail, and each failure is informative:
- If per-layer-matched random graphs do as well, the architecture-centric view of Rethinking the Value of Network Pruning extends down to circuit scale. The pruning community would learn that topology matters only at the level of budgets.
- If only exact wiring works, input-fraction atlases like those built by rabies tracing are an incomplete description of what a circuit computes. We would state that only as an analogy, not a biological claim.
Relation to Prior Work
Three circuit-discovery lines supply the topologies we test.
- ACDC prunes edges greedily backward from the output using activation patching (replacing a component's output with its output on corrupted inputs).
- DiscoGP learns differentiable masks over both edges and weights and returns self-contained "sheaves."
- HAP filters the graph with attribution scores and then runs mask-based edge pruning. It shows that cooperative units like S-inhibition heads are lost if the filter is too strict.
All three evaluate with frozen inherited weights. None asks whether the recovered graph is trainable. We reuse their outputs as the objects under test rather than proposing a new discovery method.
The manual reverse-engineering of indirect object identification in Interpretability in the Wild gives the head taxonomy we use to check whether mechanisms re-form: duplicate-token, S-inhibition, name-mover, backup and negative name-mover heads.
The compression literature motivates and constrains the design:
- Rethinking the Value of Network Pruning is the motivating result. It also warns that for unstructured sparsity the scratch advantage weakens. Circuits are closer to unstructured edge sparsity than to channel pruning, so the hypothesis is genuinely at risk.
- DropNet and Pruning from Scratch also find that pruned structures survive reinitialization.
- Non-uniform allocation work, DSA and LSA, shows that per-layer budgets alone carry much of a pruned model's quality. This is exactly why per-layer-matched random graphs are our most dangerous control.
The connectome-constrained latent model ( Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity) puts measured wiring into a dynamical model of brain activity. It never tests whether that wiring supports learning a task, which is the question we pose in silico.
Research Program
The substrate is GPT-2 Small with circuits for three tasks:
- Indirect object identification: the $p_{\text{IOI}}$ templates of Wang et al..
- Greater-Than completion, from ACDC.
- The Arithmetic track of MIB.
For each task we extract circuits with ACDC, edge attribution patching (a fast one-backward-pass gradient estimate of each edge's importance), HAP, and DiscoGP. We record the retained edges and, for DiscoGP, the retained weights.
A circuit is implemented as a GPT-2 in which each component reads only from its retained upstream edges; masked edges are zeroed at the residual-stream read. Retained weights are reinitialized with GPT-2's standard scheme, and the masks stay fixed during training.
Names and numbers only mean something through the embeddings, so we run two regimes:
- Internals-from-scratch keeps the pretrained token embedding and unembedding frozen and reinitializes everything inside. This isolates internal wiring.
- Fully-from-scratch also learns the embeddings on a restricted vocabulary.
Training uses the task's templates. A subset of templates is held out entirely to measure generalization rather than memorization.
The central object is a ladder of wiring descriptions, each coarser than the last, with random graphs sampled at each level:
- Exact wiring. The circuit itself.
- Input-profile match. Randomly rewire each retained node's inputs while preserving how many come from each upstream layer and component type (attention versus MLP). This is the in-silico analogue of a rabies input-proportion profile.
- Budget match. Preserve only the number of retained edges and parameters per layer, as in allocation methods.
- Global match. Preserve only the total count.
Mechanism recovery is scored after training. We apply the path-patching and head-classification procedure of Wang et al. and measure the fraction of original circuit heads whose role is reproduced in the same position. We run the ladder at every level with at least five seeds and ten random graphs.
Baselines
Inherited-weights circuit (upper reference). Each discovered circuit evaluated with its original weights, the standard practice in all circuit papers. For the strongest discovery method on IOI, DiscoGP reports 100% accuracy in isolation, with sheaves of 1–7% of parameters keeping 93–100% of performance. Wang et al. report a 28-head circuit that is 94% faithful. For discovery quality on MIB's IOI circuit track, EAP-IG reaches a circuit–model distance of 0.03 (MIB). This reference tells us how much of the inherited performance retraining recovers. It is not a competitor to beat.
Budget-matched random subgraph trained from scratch (strongest competitor). This is the direct prediction of Rethinking the Value of Network Pruning and the allocation results of DSA and LSA. It is strongest because it encodes the entire "shape matters, specific connections don't" hypothesis. It gets the same data, optimizer, step count, and hyperparameter sweep as the circuit, with the sweep tuned separately per condition at equal trial budget. The only reported figure for scratch versus inherited weights is from Liu et al.: on ImageNet ResNet-34, Scratch-B reaches 73.03% versus 72.56% fine-tuned. For retraining circuits from scratch, no reported result was found on Lacuna. Beating this baseline means a statistically reliable gap across seeds and graphs in both of the following:
- held-out-template accuracy or logit difference;
- examples needed to reach 90% of inherited-weights performance.
Input-profile-matched random subgraph (key ablation). This isolates whether per-node input proportions, the level at which rabies atlases describe circuits, carry the advantage. If it matches the exact circuit, exact edges are unnecessary. If it sits between exact wiring and budget match, the ladder has quantified how much function lives in edge identity.
Dense model of equal parameter count, and the naive reference. A small unmasked transformer with the circuit's parameter count, plus the untrained circuit as a floor. The dense model tests whether any capacity-matched network learns the task. It runs under the identical protocol.
Why not just…?
Why not just train a small dense transformer on IOI? This is a real threat. With enough in-template data, a dense model may learn IOI, and then the circuit adds nothing on raw accuracy. For that reason the decisive metrics are:
- held-out-template generalization;
- sample efficiency;
- mechanism recovery.
These are where an inductive bias should show. If the dense model matches the circuit on all three, we report that circuits carry no trainable bias beyond capacity.
Why not just trust Liu et al. that per-layer budgets suffice? Their evidence is for channel-level structured pruning. They also found the scratch advantage weakens for unstructured sparsity, and circuit edges resemble fine-grained sparsity. So their result does not settle the circuit case in either direction. The budget-matched control tests it directly.
Why not just keep evaluating circuits with inherited weights? Frozen-weight faithfulness cannot separate "this wiring computes the task" from "these particular weights happen to compute it on this wiring." DiscoGP reports near-perfect isolated performance, yet that number says nothing about whether the topology itself is meaningful. Retraining is the only way to test that.
Why not just use cheap attribution-patching circuits? HAP shows attribution-only circuits drop cooperative S-inhibition heads, and DiscoGP reports they fail under zero-ablation. We keep them as one of the four topology sources. We predict their topologies retrain worse, which would give a trainability-based way to rank discovery methods.
Proposed Experiments
Positive control. Tracr-compiled transformers from ACDC have known ground-truth circuits. There, exact wiring trained from scratch should clearly beat budget-matched random graphs. If it does not, the training protocol is at fault, not the hypothesis.
Main experiment. Run the full wiring ladder on IOI, Greater-Than and MIB Arithmetic, in both from-scratch regimes, for all four discovery methods.
Decisive checks. There are three:
- Does the exact circuit beat the budget-matched control on held-out templates and sample efficiency?
- Where does input-profile matching fall between the two?
- Do retrained IOI circuits re-form duplicate-token, S-inhibition and name-mover roles in the original head positions more often than chance placement predicts?
Secondary test of redundancy. Retrain circuits with the backup name-mover pathway removed versus kept. This tests whether ACDC's "OR gate" pruning of redundant parallel paths yields topologies that are less trainable or less robust, even when frozen-weight faithfulness looks identical.
References
- Rethinking the Value of Network Pruning
- Brain-wide presynaptic networks of functionally distinct cortical neurons
- Towards Automated Circuit Discovery for Mechanistic Interpretability
- Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularity
- Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small
- DropNet: Reducing Neural Network Complexity via Iterative Pruning
- Pruning from Scratch
- Discovering Sparsity Allocation for Layer-wise Pruning of Large Language Models
- LSA: Layer-wise Sparsity Allocation for Large Language Model Pruning Based on Minimal Linear Reconstruction Error
- Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity
- Connectome-constrained Latent Variable Model of Whole-Brain Neural Activity
- MIB: A Mechanistic Interpretability Benchmark