Activation Patching for Mechanistic Interpretability

1,016 concepts708 papers1 proposal

Which part of a neural network causes a behavior? Attention maps and gradients show what correlates with an output. Activation patching tests cause: run the model twice on inputs that differ in one fact, copy an internal activation from one run into the other, and see whether the output follows it. The same experiment appears under other names, including causal tracing, interchange intervention and causal mediation analysis.

Run the model on a clean prompt, "The Eiffel Tower is in the city of", and it answers Paris. Run it on a corrupted prompt, "The Colosseum is in the city of", and it answers Rome. Now rerun the corrupted prompt, but overwrite one activation, say one layer's output at the last token of the subject, with its value from the clean run. If the answer flips back to Paris, that activation carries the fact. Repeat over every layer and position and you get a map of where the information lives and when it moves.

Activation patching: the clean run answers Paris, the corrupted run answers Rome, and the corrupted run with one activation copied in from the clean run answers Paris again.

Activation patching in one picture. The patched run uses the corrupted prompt with one activation copied in from the clean run. If its answer flips back to Paris, that activation carries the fact.

Two directions, two questions

Patching clean activations into a corrupted run (denoising) asks which components are sufficient to restore the behavior. Patching corrupted activations into a clean run (noising) asks which components are necessary for it. The two can disagree, and Heimersheim and Nanda wrote the practical guide to choosing between them and reading the results.

The details of the corruption matter too. Zhang and Nanda show that corrupting with Gaussian noise or by swapping in a different subject, and scoring with probabilities or with logit differences, can lead to different conclusions about the same model. They recommend token swaps and logit differences.

Where facts live

The first result that made the method famous was about factual recall. Meng et al. corrupted the subject of a prompt with noise and restored one hidden state at a time. They called this causal tracing. Recall concentrated in middle-layer MLPs at the last token of the subject, and that pointed to where a single fact could be edited without damaging unrelated knowledge.

The same tool has been turned on reasoning. Dutta et al. patched activations during chain-of-thought and found a functional rift around the middle of the model: earlier layers move information between related entities based on pretraining, and later layers take over in-context reasoning and write the answer along several parallel paths.

From locations to circuits

Patching a whole layer tells you where information sits. To learn how it flows, path patching changes only the connection between one upstream component and one downstream component, and leaves everything else untouched.

Wang et al. used it to reverse-engineer how GPT-2 Small finishes "When John and Mary went to the store, John gave a drink to" with Mary. They found 26 attention heads in seven classes, including duplicate token heads, S-inhibition heads and name mover heads. Hanna et al. traced how the same model decides that a year must be greater than an earlier one, down to the attention heads and MLPs involved.

Finding circuits by hand takes many runs, so the search has been automated. Conmy et al. patch edges one at a time and prune those that do not matter (ACDC), and recover circuits that had been found by hand. Syed et al. replace the patching runs with a gradient-based approximation, attribution patching, which scores every edge from two forward passes and one backward pass, and finds circuits better than ACDC in their comparison.

Circuits get reused

A recurring finding is that circuits behave like reusable parts. Merullo et al. found that much of the IOI circuit also does the work in a different task, and that intervening on those shared components could repair the model's errors on the second task.

The reuse crosses languages. Ferrando et al. studied subject-verb agreement in English and Spanish and found that the model uses nearly the same circuit for both, built around a direction that encodes whether the subject is singular or plural. They report that a direction found in English moves the model's predictions in Spanish.

Tasks as vectors

Patching can also pull a task out of a model. Todd et al. patched activations across in-context examples and found a small set of attention heads that carry the task itself, for instance "give the antonym". The sum of their outputs, a function vector, can be added to an unrelated prompt and makes the model perform the task with no examples at all.

Beyond text

The experiment needs only two inputs and a way to swap activations, so it carries over to other modalities. Palit et al. adapted causal tracing to the vision-language model BLIP, corrupting the image instead of the text, and released the tool for others to use.

Momentum

The area took off after 2022. Of the papers Lacuna lists here, 20 are from 2022, 55 from 2023, 142 from 2024 and 225 from 2025. Much of the recent work scales the method up, with automated and gradient-based circuit search, or tests whether circuits found in small models hold up in large ones.

Open problems

  • Choosing the corruption. Different corrupted inputs and metrics can point to different components, and there is no agreed default for every task.
  • Necessary or sufficient. Noising and denoising answer different questions, and a component can matter in one direction and not the other.
  • Scale. Exhaustive patching grows with the number of components. Attribution patching is cheap but approximate, so its findings still need checking with real patches.
  • Generalization. A circuit found on one prompt template may not be the one the model uses on natural text.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_activation_patching_f2ebb2db,
  title = {activation patching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-patching/rdr_f2ebb2db14ec424b96173a2e4863ac48},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_activation_patching_f2ebb2db_v1,
  title = {activation patching},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/activation-patching/rdr_f2ebb2db14ec424b96173a2e4863ac48/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake