Crediting Only What Survives: Cancellation-Aware Circuit Discovery for Language Models

Most tools for explaining a language model's internals assume that each layer adds information to a shared running sum, the "residual stream", and that later layers read from it. A recent measurement breaks that assumption. In most modern open-weight models, each layer's write is partly erased by the very next layer. The outputs of consecutive layers point in opposite directions on average (cosine similarity around −0.2), and the next layer's linearized response has a sharp eigenvalue mode at −1 (LLM Layers Immediately Correct Each Other). If a layer proposes something and its neighbour deletes it, a circuit-discovery method will find two large, opposing contributions where the model's actual computation contains nothing. Explanations built this way become inflated, harder to read, and possibly less faithful, and the problem should be worst in exactly the models that matter today: Llama, Qwen, Gemma, and Mistral, as opposed to GPT-2.

We propose to change the unit of explanation. Instead of crediting each sublayer (an attention block or an MLP block) with its raw output, we credit it with its surviving write: its output plus the response that output provokes in the next few sublayers. This is cheap to compute, takes one extra partial forward pass per node, and slots directly into existing circuit-discovery and causal-localization pipelines. The claim is testable and falsifiable in a strong way. The approach should make explanations smaller at equal faithfulness in models that cancel, and it should give no benefit in GPT-2 or in OLMo-7B checkpoints saved before cancellation emerges.

Core Claim

Define $o_i$ as the output that sublayer $i$ adds to the residual stream $x_i$. Define $f_{i+1}, \dots, f_{i+k}$ as the next $k$ sublayers, and $x^{(k)}(\cdot)$ as the residual state after those $k$ sublayers when started from the given input. The surviving write of sublayer $i$ over a window of $k$ sublayers is then

$$s_i^{(k)} = o_i + \Big[x^{(k)}(x_i + o_i) - x^{(k)}(x_i)\Big] - o_i \cdot 0 = x^{(k)}(x_i + o_i) - x^{(k)}(x_i),$$

In words, it is the difference that $o_i$ makes to the stream $k$ sublayers later, holding everything else fixed. For $k=1$ and to first order, this equals $(I + J_{i+1})\,o_i$, where $J_{i+1}$ is the Jacobian of the next sublayer. The −1 eigenmode found by LLM Layers Immediately Correct Each Other is precisely the part of $o_i$ that this operator removes. Our central claim has three parts. First, a large fraction of the attribution mass that current methods assign in cancelling models sits on proposal–correction pairs that net to near zero. Second, re-expressing the computation graph in terms of surviving writes recovers circuits that are equally faithful with substantially fewer edges. Third, both effects track the presence of cancellation across model families and across training time.

This differs from the closest work in specific ways. The cancellation paper measures the phenomenon but explicitly leaves open how to separate transient "misfires" from persistent representations in interpretability tools. Residual Stream Analysis with Multi-Layer SAEs tracks features across layers by pooling residual states, but it treats layers as independent samples and never models one layer's response to another. RelP makes attribution more faithful through nonlinear residual streams, but it keeps raw sublayer outputs as the nodes. We change the nodes, not the attribution rule, so the two approaches compose.

Research Program

The first stage is measurement. We will instrument GPT-2 Small and Large (the no-cancellation controls), Llama-3-8B, Qwen2, and Gemma (the cancellation models), and a series of OLMo-7B pretraining checkpoints on both sides of the roughly 21% point where cancellation emerges. For each model we will run edge attribution patching with integrated gradients (EAP-IG), which is the top circuit-localization method in MIB: A Mechanistic Interpretability Benchmark. We will then compute the share of total edge attribution that falls on "cancelling pairs". A cancelling pair is an edge from sublayer $i$ into sublayer $i{+}1$ whose effect is opposite in sign to, and at least half the magnitude of, $i$'s direct path to the output. This stage answers whether cancellation actually contaminates circuits, which no paper in the file has checked.

The second stage builds the reparameterized graph. Each node carries its surviving write $s_i^{(k)}$ in place of $o_i$. The downstream sublayers' responses to $o_i$ are subtracted from those sublayers' own nodes, so the decomposition still sums to the same final stream. Because the correcting sublayers are nonlinear, the responses to different proposers do not add exactly. We therefore report an interaction residual, defined as the final-stream mismatch between the summed surviving writes and the true stream. We treat a large residual as a failure of the method, not as noise to hide. The window $k$ is swept over 1, 2, and 4. MLPs do most of the correcting according to the cancellation paper, so $k=1$ into the next MLP is the default. EAP-IG and RelP are then run unchanged on the new graph.

The third stage extends the same idea to causal variable localization. Distributed alignment search (a supervised method that learns a rotated subspace of activations whose swap between prompts transfers a target variable) and sparse autoencoders will each be trained on surviving writes instead of raw residual states. We then test whether the gap MIB reports between sparse-autoencoder features and neurons narrows. This stage is secondary. If stage one finds little attribution mass on cancelling pairs, we stop there and report a negative result.

Baselines

The naive reference point is EAP-IG on the standard sublayer graph. It is also the strongest existing method for circuit localization in MIB, which reports a Circuit-Model Distance of 0.03 for EAP-IG on GPT-2 IOI (Indirect Object Identification, a task where the model must complete a sentence with the name of the indirect object). We found no reported result on Lacuna for EAP-IG Circuit-Model Distance on Llama-3-8B, Qwen2, or Gemma, so we will produce these numbers ourselves with MIB's released protocol. The comparison is fair because we use the same prompts, the same integrated-gradient step count, and the same edge budgets. The only difference is the node definition. Beating EAP-IG means reaching the same Circuit Performance Ratio (the fraction of full-model task performance the circuit recovers) with at least 25% fewer edges on cancelling models. At the same time, performance on GPT-2 must be unchanged within seed variance.

The second baseline is RelP (RelP), which reports a Pearson correlation of 0.956 with exact activation patching on GPT-2 Large MLP outputs for IOI. By comparison, plain attribution patching reaches only 0.006. RelP is the strongest competitor on the question of faithful attribution through nonlinear residual streams. We will run RelP on both graphs and use exact activation patching as ground truth. Here the question is whether surviving-write nodes preserve RelP's correlation while shrinking circuits.

The key-ingredient ablation is adjacent-layer merging. Sublayers $i$ and $i{+}1$ are fused into one node, so the correction is absorbed without being attributed to anyone. This isolates what our method adds: it separates the next layer's response to $o_i$ from that layer's own independent computation. A random-window control assigns a sublayer's response to a randomly chosen non-adjacent proposer. It checks that gains come from the actual correction structure and not from the mere act of reparameterizing.

Why not just…?

Why not just merge each pair of adjacent layers into one node? This is the most serious threat. It could plausibly capture most of the benefit, because cancellation is dominated by the immediately next MLP. It loses resolution, however. The next layer also performs its own computation, and merging credits that computation to the proposer's node, which halves the graph's granularity everywhere, including the roughly 80% of directions that are not cancelled. The merged-node ablation above settles the question directly. If merging matches surviving writes on edges-at-equal-faithfulness, we will recommend merging, the simpler tool, and report that the finer decomposition bought nothing.

Why not just trust EAP-IG, since integrated gradients already measure total effect on the output? Total-effect attribution does give the net of a proposal–correction pair the right value. The problem is that it assigns large, opposite scores to the two edges separately, and top-k edge selection then keeps both. The file contains no measurement of how often this happens. Stage one produces that number. If cancelling pairs carry a negligible share of attribution mass in Llama or Qwen, this alternative wins and the project ends with that finding.

Why not just train sparse autoencoders on the residual stream every other layer, or only after the correcting MLP? A residual state taken after the correction already excludes that correction's misfire. The cancellation paper's diagnosis that sparse-autoencoder features capture transient misfires would then apply only to states between proposal and correction. This is cheap and should be run as the baseline for stage three. It cannot, however, say which component produced a surviving feature, and that attribution is what circuit discovery needs. Its IIA (interchange-intervention accuracy: how often swapping the learned representation between prompts transfers the target variable) will be compared with ours on MIB's causal variable track. The reference points there are sparse autoencoders at 50–70% and distributed alignment search at 94–100% on ARC (MIB).

Decisive Experiments

The primary benchmark is MIB circuit localization on IOI and ARC. Metrics are Circuit Performance Ratio and Circuit-Model Distance swept over edge budgets. The headline number is the number of edges needed to reach a fixed performance ratio, with exact activation patching as the faithfulness reference. Secondary checks are Subject-Verb Agreement, as used by RelP, and RAVEL entity-attribute disentanglement (RAVEL), where HyperDAS reports a Disentangle score of 84.7 (HyperDAS) as the upper reference.

Two checks decide the project. The first is the family control. The edge reduction must appear in Llama, Qwen, and Gemma and be absent in GPT-2 and Phi, which lack cancellation according to LLM Layers Immediately Correct Each Other. The second is the within-model temporal control. In OLMo-7B, the reduction must be near zero at checkpoints before about 0.5 trillion tokens and grow afterwards, in step with the measured strength of cancellation. The temporal control holds architecture and data fixed, so a gain that appears without cancellation, or that fails to track it, falsifies the mechanism. The interaction residual must also stay small, under 10% of the final-stream norm at $k=1$. Otherwise the decomposition is not an explanation of the model at all.

Finally, we will ask whether the IOI circuit of Interpretability in the Wild, which was found in GPT-2 Small and relies mainly on attention heads, has a recognizable counterpart in Llama-3-8B once cancelling MLP pairs are factored out. That paper's "negative name mover" heads suppress the correct answer. We will check whether, in larger models, this suppressive role migrates into the MLP corrections that the cancellation paper attributes to feed-forward blocks.