Causal Probing for Functional Utility in Language Models

This direction is split into other directions: probing-causality gap. Its last version stays readable and citable here.

Version 1 of 145 papers51 conceptsfrozen 2026-10-04

For years, the standard way to understand what a neural network "knows" was to use a diagnostic probe. If you want to know if BERT understands plural nouns, you take its internal activations, train a simple linear classifier (the probe) to predict "singular" or "plural," and if the classifier is accurate, you conclude the model has learned grammatical number.

However, as James R. Glass and Yonatan Belinkov point out in their comprehensive survey, there is a fundamental logical gap in this approach: extractability does not imply utility. Just because an external observer can find a piece of information hidden in a model's activations doesn't mean the model itself is actually using that information to make its predictions.

The Correlation-Causation Gap

The problem with traditional probing is that it only establishes a correlation. A model might encode "grammatical number" as a byproduct of some other statistical pattern, yet ignore that encoding when it actually comes time to conjugate a verb.

Researchers like Yanai Elazar and Shauli Ravfogel demonstrated this "decoupling" in their work on Amnesic Probing. They found that even if you surgically remove a highly extractable linguistic feature from a model's representation, the model’s performance often remains unchanged. This suggests that the information we "find" via probing is often incidental—a "latent" property that isn't functionally causal.

Surgical Interventions as Proof

To bridge this gap, a new wave of research has moved from observation to intervention. The goal is to prove functional utility: if we change the internal representation of a specific concept, does the model's output change in a predictable, causal way?

One of the most effective methods for this is AlterRep, developed by Shauli Ravfogel, Tal Linzen, and Yoav Goldberg. In their study on subject-verb agreement, they didn't just look for syntactic boundaries; they surgically altered them. By "pushing" the embeddings in a way that neutralized the representation of a relative clause, they could directly observe the model's agreement logic break down, proving that those specific representations were functionally necessary.

Similarly, Sophie Hao and Tal Linzen showed that verb conjugation in Transformers is driven by a linear encoding of subject number. By identifying a specific "number subspace" and flipping the vectors across it, they could "trick" the model into switching from singular to plural conjugations with near-perfect success. This is the "gold standard" of interpretability: a causal intervention that results in a controlled behavioral change.

Mapping the Functional Flow

This causal approach allows us to map not just where information is, but how it flows through the model. Karim Lasri and Ryan Cotterell used these techniques to probe for the usage of grammatical number, showing that models use distinct linear subspaces for nouns and verbs and that this information is actively leveraged at specific layers.

The implications extend beyond syntax. In multilingual models, Kyle Mahowald and colleagues used counterfactual probing to show that "language identity" is a functional feature. By steering an embedding toward a different language's subspace, they could force the model to predict words from that target language, proving that the model's internal "language tag" is a primary driver of its vocabulary selection.

Why This Matters

The shift from diagnostic probing to causal intervention represents a maturation of the field. As Ananth Agarwal and Christopher D Manning argue in recent work, failing to distinguish between mechanisms and outcomes leads to a false sense of understanding. We might think we've "solved" how a model understands syntax because our probes are accurate, while the model is actually relying on entirely different, non-syntactic heuristics.

By demanding causal evidence, researchers are moving past "peeking inside the box" to actually "rewiring the box." This doesn't just tell us what the model knows; it tells us how the model works.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_causal_probing_for_functional_utility_in_25c95ad8,
  title = {Causal Probing for Functional Utility in Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-probing-for-functional-utility-in-language-models/rdr_25c95ad833b14627ac8047cf4fa6ca50},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_causal_probing_for_functional_utility_in_25c95ad8_v1,
  title = {Causal Probing for Functional Utility in Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/causal-probing-for-functional-utility-in-language-models/rdr_25c95ad833b14627ac8047cf4fa6ca50/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}