Abstract
Mechanistic interpretability aims to reverse-engineer neural networks into human-understandable circuits. The current "gold standard" for this task is counterfactual intervention—asking whether an output changes if a specific activation is patched or ablated. This paper presents a theoretical and empirical critique of this approach, arguing that it systematically fails to account for overdetermination (redundancy) and non-transitivity in causal chains. By applying these concepts to a case study of "succession" behavior in the Pythia 70M model, we show that standard methods miss critical "backup" mechanisms and fail to map the hierarchical composition of features.
Introduction
In the quest to understand the inner workings of large language models (LLMs), researchers have increasingly turned to mechanistic interpretability. The goal is to identify "circuits"—subgraphs of the model that perform specific tasks. To validate these circuits, the field relies heavily on counterfactual theories of causality: if changing activation $A$ changes output $Y$, then $A$ is considered a cause of $Y$.
While counterfactual patching is more robust than simple correlation, it inherits classic philosophical traps. This paper explores two primary challenges: overdetermination, where multiple redundant features can produce the same effect, and non-transitivity, where local causal steps do not aggregate into a global causal dependency.
The Problem of Overdetermination
A major hurdle for counterfactual methods is redundancy. In neural networks, multiple neurons or "heads" often perform the same function—a phenomenon known as "backup heads." If two features, $A_1$ and $A_2$, both independently cause $Y$, ablating either one individually will not change $Y$. Consequently, a standard counterfactual test would conclude that neither $A_1$ nor $A_2$ is a cause, leading to a significant loss in recall for circuit discovery.
As shown in Figure 3, overdetermination (disjunction) creates a scenario where the output $y$ is stable even if one of its causes is removed. This "leaky" logic means that current interpretability tools, which primarily focus on "breaking" things (noising), are biased toward finding bottlenecks rather than the full, robust logic of the model.
Non-Transitivity and the Composition Problem
The second challenge is the non-transitivity of counterfactual dependence. In a simple chain $A \to B \to y$, it is often assumed that $A$ is a cause of $y$. However, in many systems, $y$ may depend on $B$, and $B$ on $A$, yet $y$ does not counterfactually depend on $A$.
This is illustrated in Figure 1. This "Hiker Fallacy" (where a hiker ducks to avoid a boulder and survives, yet the boulder is not considered the "cause" of survival because they would have lived anyway) suggests that we often miss the "triggers" of behavior. In neural networks, high-level abstractions are built from low-level detectors. If we only look for features that directly flip the final output logit, we miss the entire "middle management" of the network—the features that compose into the final decision-making units but aren't strictly necessary for the final output to exist in some state.
Case Study: Succession in Pythia 70M
To demonstrate these theoretical points, the paper examines a "succession" behavior in the Pythia 70M model—the ability to predict the next item in a sequence like "1, 2, 3." Using Sparse Autoencoders (SAEs) to find interpretable features, the author attempted to map the circuit for this behavior.
Standard circuit discovery methods suggested that a "successor feature" in layer 2 appeared almost spontaneously. However, by moving beyond global counterfactuals and looking at local dependencies (i.e., "denoising" or injecting features), the author discovered a rich underlying structure. The successor feature was actually composed of specific number detectors and list-context detectors from earlier layers. These earlier features were "missed" by standard patching because they were either redundant or their influence was non-transitive relative to the final output logit.
Conclusion and Future Directions
The reliance on counterfactual "noising" creates a Pareto frontier between the simplicity of an explanation and its recall. To move toward more faithful interpretations, the author suggests:
- Denoising Interventions: Instead of just breaking features, researchers should try injecting them into contexts where they are absent to find redundant causes.
- Local Dependency Mapping: We must trace how features compose into other features, rather than just how they affect the final output.
- Benchmarking with Redundancy: The community needs "toy models" with known redundant mechanisms to evaluate how well new interpretability tools can recover complex, non-bottleneck logic.
While mapping every local dependency risks an explosion of complexity, it is a necessary step to move from "plausible stories" to a truly mechanistic understanding of how neural networks compute.
Summary updated