Causal Interventions for Neural Representation Analysis

22 concepts19 papers1 proposal

For years, the standard tool for peering inside the "black box" of language models was the diagnostic probe. If we wanted to know whether a model like BERT understood syntax, we trained a simple, supervised classifier on top of its frozen internal representations to predict grammatical structures, such as parse trees (Hewitt et al.). If the probe achieved high accuracy, researchers concluded that the model encoded that linguistic knowledge.

However, a fundamental methodological challenge soon emerged: the distinction between the presence of information and its functional utility (Glass et al.). As Movva et al. observed, a probe can successfully extract a feature from a model's layers without providing any guarantee that the model actually leverages that feature during inference. This discrepancy is known as the probing-causality gap.

Why the Probing-Causality Gap Occurs

To understand why traditional probing fails to establish causation, consider what a probe actually does. It is an external, optimized model designed to find patterns. Because deep neural networks are massive statistical correlation engines, their hidden states contain a vast array of passive data storage and read-only artifacts (Andreas et al.).

For example, language models implicitly encode highly detailed character-level spelling information within their token embeddings (Itzhak et al.) and representations of future planning states (Hanna et al.). Yet, the presence of these features does not mean they actively drive downstream reasoning or generation. A probe might detect a strong correlation between a representation and a linguistic property, but the model's internal attention mechanism may completely ignore it (Mori et al., Miaschi et al.). Furthermore, fine-tuning a model can significantly alter or increase the presence of structural features without those features acting as the primary drivers of final performance (Sarti et al., Alleman et al.).

Moving from Observation to Causal Intervention

To prove that a feature is functionally necessary, interpretability research has shifted from passive observation to active, causal intervention. The intuition is straightforward: if you surgically remove or alter a specific piece of information within a representation, and the model's downstream behavior changes in a predictable way, you have established a causal link.

Amnesic Probing and Subspace Erasure

One way to test functional utility is to erase a concept entirely. Elazar et al. introduced Amnesic Probing, a method that uses causal interventions to drop specific linguistic properties from representations. By systematically erasing features like part-of-speech tags, the authors demonstrated that high probing performance often fails to correlate with a property's actual utility for the model's final predictions.

This erasure is possible because linguistic features are often organized within low-dimensional linear subspaces of the model's high-dimensional vector space (Andreas et al.). By projecting hidden states orthogonal to these specific dimensions, researchers can "blind" a model to a precise grammatical rule while leaving the rest of its capabilities intact.

Counterfactual Manipulations

Going beyond erasure, researchers can actively manipulate representations to "trick" the model. Ravfogel et al. developed AlterRep, a framework that surgically alters internal representations of relative clauses. By modifying these boundaries, the authors provided direct causal evidence showing exactly when and where a model relies on syntactic structure to predict subject-verb agreement.

Similarly, Hao et al. isolated a linear "number subspace" that encodes grammatical number (singular vs. plural). By flipping the vectors across this geometric plane, they causally forced the model to switch its verb conjugations with near-total success, proving that this specific linear encoding is the functional mechanism driving the behavior.

Systematizing Causal Interpretability

As the field matures, ad-hoc interventions are giving way to systematic benchmarks and structured architectures. Pandey demonstrated that syntax-guided neural module distillation can explicitly trace whether features are actively utilized during the representation construction process rather than just appearing as passive byproducts.

To evaluate these methods uniformly, Arora et al. developed CausalGym, a benchmark for testing causal interpretability methods on linguistic tasks. Their evaluations show that advanced intervention techniques, such as Distributed Alignment Search (DAS), can successfully locate and manipulate the precise internal "levers" controlling a model's output. These causal tools allow researchers to map out fine-grained functional mechanics, such as how multilingual models allocate entirely different sets of neurons to handle subject-verb agreement depending on whether the target words are adjacent or separated by clauses (Mueller et al.).

Key Foundational Work

Analysis Methods in Neural Language Processing: A Survey (2019) — contributed 2 concept elements to this direction.
When Bert Forgets How To POS: Amnesic Probing of Linguistic Properties and MLM Predictions (2020) — contributed 1 concept elements to this direction.
Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_probing_causality_gap_dece408c,
  title = {probing-causality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probing-causality-gap/rdr_dece408c5273418ab5d1fea8e81783df},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_probing_causality_gap_dece408c_v1,
  title = {probing-causality gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/probing-causality-gap/rdr_dece408c5273418ab5d1fea8e81783df/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake