TL;DR
Standard mechanistic interpretability often assumes a neural network implements a single, uniform algorithm across all inputs, leading to causal abstractions that are only partially faithful. This paper introduces a method to combine multiple candidate causal models by dynamically partitioning the input space based on which computational pathway the model executes. By evaluating GPT-2 small on arithmetic and Boolean logic tasks, the authors demonstrate that combined causal models achieve significantly higher explanatory strength at strict faithfulness thresholds than any single model alone.
What Problem Does This Paper Address?
In mechanistic interpretability, causal abstraction provides a rigorous mathematical framework to prove whether a neural network realizes a specific high-level algorithm. Researchers align intermediate variables of an algorithmic causal model with internal activations (or linear subspaces) of the neural network using interchange interventions—counterfactually swapping internal representations between different inputs to see if the network behaves as the algorithm predicts.
However, in practice, candidate causal abstractions are rarely 100% faithful; they often achieve only intermediate Interchange Intervention Accuracy (IIA), such as 60–80%. This occurs because deep neural networks do not discretely transition between computational steps, nor do they process every input identically. A model might evaluate small numbers sequentially while computing larger combinations in parallel.
This creates a fundamental dilemma:
- Trivial faithfulness: An abstract model with no internal intermediate variables trivially achieves 100% faithfulness, but explains nothing (it leaves the network a black box).
- Hypothesis strength vs. Faithfulness trade-off: Hypotheses asserting detailed intermediate computations make strong claims, but risk failing on subsets of inputs.
The authors address how to formalize and navigate this trade-off to construct faithful, non-trivial mechanistic explanations.
Methods
The authors propose dynamically changing the high-level causal structure conditioned on the input:
Input x ───► [ Input Space Partition ] ───► Route to Model M_j (if x ∈ Δ_j)
└──► Route to Trivial Model (if unassigned)
- Combined Causal Models: Given candidate models $M^1, \dots, M^k$ sharing the same inputs and outputs but having different intermediate variables $V_1, \dots, V_k$, a combined model $M^*$ partitions the input space into subsets $\Delta_1, \dots, \Delta_k$. When an input falls into partition $\Delta_j$, the mechanisms for $V_j$ are active, while variables from other candidate models are assigned a null value ($\emptyset$).
- Evaluation Graphs: For each candidate model $M^j$, the authors construct an evaluation graph where nodes represent valid inputs. An edge between nodes $u$ and $v$ is weighted ($0, 0.5,$ or $1.0$) based on whether interchange interventions succeed when swapping representations between base $u$ and source $v$.
- Greedy Input-Space Partitioning: Given a desired faithfulness threshold $\lambda$ (e.g., $\lambda = 0.95$ IIA), a greedy graph-pruning algorithm identifies the largest subset of unassigned inputs that maintains an average edge weight of at least $\lambda$. The remaining unassigned inputs are routed to a "trivial" model (an input-output map without internal variables).
- Alignment via Distributed Alignment Search (DAS): Using the
pyvenelibrary, intermediate variables of each candidate model are aligned to low-dimensional linear subspaces ($k \in \{64, 128, 256\}$ dimensions) within the transformer's residual stream.
Datasets
The authors designed two controlled, synthetic sequence-classification tasks using fine-tuned GPT-2 small:
- 3-Number Addition: Prompts of the form
X+Y+Z=where $X, Y, Z \in \{1, \dots, 10\}$, totaling $10^3 = 1{,}000$ unique inputs. GPT-2 was fine-tuned on 2,560 examples to achieve 100% factual accuracy. - Boolean Logic Expressions: Prompts of the form
OP1 (OP2(X) B OP3(Y)) =, where $X, Y \in \{\text{True}, \text{False}\}$, unary operators $\text{OP}_i \in \{\neg, I\}$ (negation or identity), and binary operators $B \in \{\land, \lor\}$, yielding $2^6 = 64$ exhaustive input combinations.
Key Insights and Ideas
- Smooth Algorithmic Blending: Probing layers across GPT-2 reveals that intermediate layers do not jump abruptly between states. In addition, early layers represent individual variables $(X, Y, Z)$ separately, while middle layers (layers 5–10) partially represent multiple intermediate sums ($X+Y$, $Y+Z$, or $X+Z$) simultaneously.
- Asymmetric Operations in Causal Attention: For addition, intermediate states computing $X+Y$ first were far more accurate than $Y+Z$ or $X+Z$. Because of causal masking, tokens at position $Y$ can attend to $X$, but not to $Z$.
- Hypothesis Strength: The paper formalizes strength as the proportion of inputs assigned to non-trivial models (models with internal variables). A combined model allows researchers to tune the balance: accept slightly weaker overall claims in exchange for near-perfect causal faithfulness on the inputs that are explained.
Key Results
- Arithmetic Task (Layer 7):
- When requiring high faithfulness thresholds ($\lambda \in [0.90, 1.00]$), individual candidate models (such as $M^{XY}$ or $M^{Z}$) explain relatively small fractions of the input space.
- Combining models substantially increases strength: combining $M^{Y}$ and $M^{XY}$ or combining all partial models ($M^X + M^Y + M^Z + M^{XY} + M^{XZ} + M^{YZ}$) covers significantly more inputs at $\ge 0.95$ faithfulness.
- At lower thresholds ($\lambda \le 0.80$), performance converges because individual models alone meet the permissive target.
| Model Strategy | Faithfulness Threshold ($\lambda$) | Interpretable Input Space (%) |
|---|---|---|
| Boolean Task (Layer 10) | ||
| Individual models ($M^X$, $M^{OP_2}$) | $1.00$ | $\le 50\%$ |
| Combined model ($M^{X + X'}$) | $1.00$ | $75\%$ |
| Combined model ($M^{OP_2 + V}$) | $1.00$ | $88\%$ |
Limitations and Future Work
- Toy Task Complexity: The framework was tested exclusively on small synthetic datasets (1,000 addition instances and 64 Boolean expressions) and small models (GPT-2 small). Scaling the construction of evaluation graphs ($O(|V|^2)$ pair evaluations) to open-domain natural language prompts is computationally demanding.
- Heuristic Partitioning: The input partitioning relies on a greedy subgraph search based on node degree. The authors note that more sophisticated optimization criteria could yield tighter, more generalizable partitions.
- Alignment Independence: Alignments for candidate models are learned independently rather than jointly optimized alongside the input partitions. Combining the alignment search directly with partition discovery remains an open area for future research.
Summary updated