Neural networks are often described as "black boxes," but a growing body of research suggests they are more like complex clockwork. By using a technique called causal mediation analysis (or activation patching), researchers are moving beyond simply observing what a model knows to identifying exactly where and how that information is processed.
The Intuition of Corrupt and Restore
The fundamental challenge in understanding a model like GPT-4 is that "decodability" does not equal "causality." As Lianghuan Huang and Yingshan Chang point out in Causality ≠ Decodability, and Vice Versa, a model might represent a concept in its hidden states without actually using that information to calculate its final answer.
To find the components that actually matter, researchers use a "corrupt and restore" methodology. Imagine a model trying to complete the sentence "The capital of France is..." 1. The Clean Run: The model correctly predicts "Paris." 2. The Corrupted Run: Researchers add noise to the input embeddings (e.g., blurring the word "France") so the model can no longer predict "Paris." 3. The Patching Intervention: Researchers surgically take a specific activation (like the output of a single MLP layer or an attention head) from the "Clean Run" and "patch" it into the "Corrupted Run."
If patching that specific component restores the model's ability to say "Paris," we have causal evidence that this component is a critical bottleneck for that specific fact. Fred Zhang and Neel Nanda have formalized these Best Practices of Activation Patching, advocating for symmetric token replacement to ensure the interventions are as precise as possible.
Localizing Knowledge in the MLP Layers
One of the most striking discoveries using this method is that factual knowledge is not smeared uniformly across the network. In Locating and Editing Factual Associations in GPT, Kevin Meng, David Bau, and colleagues found that facts are stored in "early sites"—specifically middle-layer feed-forward networks (MLPs).
These MLPs act as linear associative memories. For example, when the model sees "Eiffel Tower," these layers "enrich" the subject representation with the concept of "Paris." This discovery led to ROME (Rank-One Model Editing), which allows researchers to update a specific fact (e.g., making the model believe the Eiffel Tower is in Rome) by surgically modifying only a few dozen neurons.
However, this "Knowledge Neuron" view is not without controversy. Jingcheng Niu and Zining Zhu argue in What does the Knowledge Neuron Thesis Have to do with Knowledge? that these neurons are often multi-functional, governing both facts and grammar, making surgical edits more brittle than they first appear.
The Three Stage Pipeline of Reasoning
Beyond static facts, causal mediation has revealed how models perform multi-step reasoning. Researchers like Zijian Wang have identified a three-stage pipeline for relational concepts:
- Subject Enrichment: Early MLPs identify the subject.
- Relational Emergence: Middle layers represent the abstract relation (e.g., "capital of") independently of the subject.
- Answer Extraction: Late-layer attention heads move this information to the final token to produce the output.
This modularity extends to complex logic. Guan Zhe Hong and Rina Panigrahy mapped circuits for propositional logic, finding specialized families of attention heads that act as "rule-finders" and "information-movers." Similarly, Subhabrata Dutta and Tanmoy Chakraborty discovered a "functional rift" in Chain-of-Thought reasoning, where the model's first half focuses on gathering context and the second half shifts to parallel "answer-writing."
Cross-Modal Bottlenecks and Bias
The "causal lens" is also being applied to Vision-Language Models (VLMs). Alessandro Serra and colleagues identified a "Narrow Gate" where visual information is injected into the text stream through a very small number of specific tokens. This explains why models often struggle with charts; as Alexa R. Tartaglini found, the bottleneck isn't the model's "eyesight" but the vision-language handoff where the language module fails to retrieve the visual data correctly.
Finally, this localization allows us to find and mitigate harms. Arnab Sen Sharma and David Bau used activation patching to locate demographic bias in healthcare LLMs, finding that gender information is highly localized in specific MLP layers. By identifying these causal sites, we can perform "surgical interventions" to ensure models treat patients fairly without needing to retrain the entire system.