Imagine training a neural network to play chess. Seeking to understand its internal logic, you train a simple linear classifier—a probe—on the network's internal activations. To your excitement, the probe decodes the exact position of every piece on the board with near-perfect accuracy. It feels like a triumph for interpretability; you conclude that the model has constructed an internal "world model" of the game.
But then, a paradox occurs: when faced with a critical turn, the model makes a completely illegal move.
How can a model perfectly "know" the state of its environment, yet act as if it doesn't? This scenario illustrates the probing-causality gap: the fundamental disconnect between the information that can be decoded from a model's internal representations and the information the model actually utilizes to compute its final output.
In mechanistic interpretability, distinguishing whether an internal feature is a functional gear driving the model's behavior or merely a passive byproduct of training is one of the most critical challenges.
The Illusion of Decodability
For years, representational probing has been a foundational tool in interpretability. The core assumption is intuitive: if a linear probe can extract a concept (like grammatical syntax or a board state) from a model's hidden layers, then the model must be using that concept.
However, probing only establishes correlation, not causation. Because deep neural networks are highly efficient feature extractors, they frequently encode rich structural information about their inputs simply as a statistical byproduct of training.
This exact failure mode was demonstrated by András Balogh et al. in Verification of the Implicit World Model in a Generative Model via Adversarial Sequences. By testing language models on adversarial chess sequences, the authors found that a model can maintain a highly accurate internal representation of the board state in its hidden layers while simultaneously generating moves that completely violate the rules of the game. Similarly, Abhinav Joshi et al. showed in CheckersGPT: Learning World Models through Language Modeling that while linear probes readily find game-state abstractions inside a model, this observation alone fails to prove a causal link to functional decision-making.
Just because information is accessible to an external observer does not mean it is accessible or relevant to the model's downstream layers.
Passive Bystanders vs. Functional Gears
When we look inside a model, we often find beautiful geometric structures. For instance, Jonathan Mamou et al. showed in Emergence of Separable Manifolds in Deep Language Representations that language models naturally untangle complex data into linearly separable manifolds. Yet, as Michael I. Ivanitskiy et al. caution in Linearly Structured World Representations in Maze-Solving Transformers, distinguishing between these functional world models and incidental data correlations remains an open challenge.
This problem extends down to the level of individual neurons. An interpretability method might generate a highly plausible, human-readable explanation for what a specific neuron detects. However, Jing Huang et al. revealed a stark reality in Rigorously Assessing Natural Language Explanations of Neurons: even high-confidence, automated explanations of neuron activations often show nearly zero causal influence on the model's actual behavior. The neuron might be a "passive bystander"—firing perfectly when a concept appears, but passing a signal that the rest of the network ultimately ignores.
As a single author, Zaki Alaoui formalizes this distinction in Causal Interpretation of Neural Network Computations with Contribution Decomposition, noting that a significant gap exists between understanding what features a neuron represents and determining which of those representations are actually utilized to drive specific classifications.
Bridging the Gap via Counterfactual Interventions
To cross the probing-causality gap, researchers are moving away from purely observational methods and toward active, counterfactual interventions. The intuition is simple: if an internal representation is truly a functional gear, then surgically altering that representation must predictably change the model's behavior.
Observational Probing (Correlational):
Input Data ──> [ Hidden Layer (Probe Decodes Concept) ] ──> Output Move
│
└──> Correlation found, but does it drive the output?
Interventional Patching (Causal):
Input Data ──> [ Hidden Layer (Surgically Alter Concept) ] ──> Altered Output Move
│
└──> If output changes predictably, the representation is causal.
Several recent works have successfully used this interventional paradigm to prove causality:
- Grammatical Steering: Tiwalayo Eisape et al. applied counterfactual probing in Probing for Incremental Parse States in Autoregressive Language Models, demonstrating that surgically manipulating hidden states could actively steer a model's grammatical expectations.
- Entity Binding: In Representational Analysis of Binding in Language Models, Qin Dai et al. identified a low-rank "Ordering ID" subspace. By mathematically shifting activations along this direction, the authors causally forced the model to swap its internal associations between entities.
- Language-Agnostic Concepts: Clément Dumas et al. utilized activation patching in Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformers to isolate and manipulate abstract semantic concepts independently of the surface-level language being used.
By forcing a change in the latent space and observing the downstream effects, these techniques verify functional necessity rather than just representational presence.
The Disconnect Between Detection and Control
Even when we can intervene, a final nuance remains: the features that are easiest to detect are not always the easiest to control.
In AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders, Zhengxuan Wu et al. evaluated how effectively different interpretability methods could steer concepts. They discovered that dictionary learning methods optimized purely for isolating and detecting features (like Sparse Autoencoders) often underperform simpler baselines when it comes to actually modifying model behavior.
This underscores the core lesson of the probing-causality gap: optimizing for linear separability or feature detection does not automatically grant the causal leverage required to control a model. True understanding requires looking past what a model remembers to isolate the precise mechanisms it uses.
Key Foundational Work
On the relationship between visual attributes and convolutional networks (2015) — contributed 1 concept elements to this direction.
Revisiting the Importance of Individual Units in CNNs via Ablation (2018) — contributed 1 concept elements to this direction.
Influence-Directed Explanations for Deep Convolutional Networks (2018) — contributed 1 concept elements to this direction.