Combining Causal Models for More Accurate Abstractions of Neural Networks

Theodora-Mara Pîslar, Sara Magliacane, Atticus Geiger

TL;DR

Standard mechanistic interpretability often assumes a neural network implements a single, uniform algorithm across all inputs, leading to causal abstractions that are only partially faithful. This paper introduces a method to combine multiple candidate causal models by dynamically partitioning the input space based on which computational pathway the model executes. By evaluating GPT-2 small on arithmetic and Boolean logic tasks, the authors demonstrate that combined causal models achieve significantly higher explanatory strength at strict faithfulness thresholds than any single model alone.

What Problem Does This Paper Address?

In mechanistic interpretability, causal abstraction provides a rigorous mathematical framework to prove whether a neural network realizes a specific high-level algorithm. Researchers align intermediate variables of an algorithmic causal model with internal activations (or linear subspaces) of the neural network using interchange interventions—counterfactually swapping internal representations between different inputs to see if the network behaves as the algorithm predicts.

However, in practice, candidate causal abstractions are rarely 100% faithful; they often achieve only intermediate Interchange Intervention Accuracy (IIA), such as 60–80%. This occurs because deep neural networks do not discretely transition between computational steps, nor do they process every input identically. A model might evaluate small numbers sequentially while computing larger combinations in parallel.

This creates a fundamental dilemma:

  • Trivial faithfulness: An abstract model with no internal intermediate variables trivially achieves 100% faithfulness, but explains nothing (it leaves the network a black box).
  • Hypothesis strength vs. Faithfulness trade-off: Hypotheses asserting detailed intermediate computations make strong claims, but risk failing on subsets of inputs.

The authors address how to formalize and navigate this trade-off to construct faithful, non-trivial mechanistic explanations.

Methods

The authors propose dynamically changing the high-level causal structure conditioned on the input:

Input x ───► [ Input Space Partition ] ───► Route to Model M_j (if x ∈ Δ_j)
                                       └──► Route to Trivial Model (if unassigned)
  1. Combined Causal Models: Given candidate models $M^1, \dots, M^k$ sharing the same inputs and outputs but having different intermediate variables $V_1, \dots, V_k$, a combined model $M^*$ partitions the input space into subsets $\Delta_1, \dots, \Delta_k$. When an input falls into partition $\Delta_j$, the mechanisms for $V_j$ are active, while variables from other candidate models are assigned a null value ($\emptyset$).
  2. Evaluation Graphs: For each candidate model $M^j$, the authors construct an evaluation graph where nodes represent valid inputs. An edge between nodes $u$ and $v$ is weighted ($0, 0.5,$ or $1.0$) based on whether interchange interventions succeed when swapping representations between base $u$ and source $v$.
  3. Greedy Input-Space Partitioning: Given a desired faithfulness threshold $\lambda$ (e.g., $\lambda = 0.95$ IIA), a greedy graph-pruning algorithm identifies the largest subset of unassigned inputs that maintains an average edge weight of at least $\lambda$. The remaining unassigned inputs are routed to a "trivial" model (an input-output map without internal variables).
  4. Alignment via Distributed Alignment Search (DAS): Using the pyvene library, intermediate variables of each candidate model are aligned to low-dimensional linear subspaces ($k \in \{64, 128, 256\}$ dimensions) within the transformer's residual stream.

Datasets

The authors designed two controlled, synthetic sequence-classification tasks using fine-tuned GPT-2 small:

  • 3-Number Addition: Prompts of the form X+Y+Z= where $X, Y, Z \in \{1, \dots, 10\}$, totaling $10^3 = 1{,}000$ unique inputs. GPT-2 was fine-tuned on 2,560 examples to achieve 100% factual accuracy.
  • Boolean Logic Expressions: Prompts of the form OP1 (OP2(X) B OP3(Y)) =, where $X, Y \in \{\text{True}, \text{False}\}$, unary operators $\text{OP}_i \in \{\neg, I\}$ (negation or identity), and binary operators $B \in \{\land, \lor\}$, yielding $2^6 = 64$ exhaustive input combinations.

Key Insights and Ideas

  • Smooth Algorithmic Blending: Probing layers across GPT-2 reveals that intermediate layers do not jump abruptly between states. In addition, early layers represent individual variables $(X, Y, Z)$ separately, while middle layers (layers 5–10) partially represent multiple intermediate sums ($X+Y$, $Y+Z$, or $X+Z$) simultaneously.
  • Asymmetric Operations in Causal Attention: For addition, intermediate states computing $X+Y$ first were far more accurate than $Y+Z$ or $X+Z$. Because of causal masking, tokens at position $Y$ can attend to $X$, but not to $Z$.
  • Hypothesis Strength: The paper formalizes strength as the proportion of inputs assigned to non-trivial models (models with internal variables). A combined model allows researchers to tune the balance: accept slightly weaker overall claims in exchange for near-perfect causal faithfulness on the inputs that are explained.

Key Results

  • Arithmetic Task (Layer 7):
  • When requiring high faithfulness thresholds ($\lambda \in [0.90, 1.00]$), individual candidate models (such as $M^{XY}$ or $M^{Z}$) explain relatively small fractions of the input space.
  • Combining models substantially increases strength: combining $M^{Y}$ and $M^{XY}$ or combining all partial models ($M^X + M^Y + M^Z + M^{XY} + M^{XZ} + M^{YZ}$) covers significantly more inputs at $\ge 0.95$ faithfulness.
  • At lower thresholds ($\lambda \le 0.80$), performance converges because individual models alone meet the permissive target.
Model Strategy Faithfulness Threshold ($\lambda$) Interpretable Input Space (%)
Boolean Task (Layer 10)
Individual models ($M^X$, $M^{OP_2}$) $1.00$ $\le 50\%$
Combined model ($M^{X + X'}$) $1.00$ $75\%$
Combined model ($M^{OP_2 + V}$) $1.00$ $88\%$

Limitations and Future Work

  • Toy Task Complexity: The framework was tested exclusively on small synthetic datasets (1,000 addition instances and 64 Boolean expressions) and small models (GPT-2 small). Scaling the construction of evaluation graphs ($O(|V|^2)$ pair evaluations) to open-domain natural language prompts is computationally demanding.
  • Heuristic Partitioning: The input partitioning relies on a greedy subgraph search based on node degree. The authors note that more sophisticated optimization criteria could yield tighter, more generalizable partitions.
  • Alignment Independence: Alignments for candidate models are learned independently rather than jointly optimized alongside the input partitions. Combining the alignment search directly with partition discovery remains an open area for future research.

Summary updated

Abstract

Mechanistic interpretability aims to reverse engineer neural networks by uncovering which high-level algorithms they implement. Causal abstraction provides a precise notion of when a network implements an algorithm, i.e., a causal model of the network contains low-level features that realize the high-level variables in a causal model of the algorithm (Geiger et al., 2024). A typical problem in practical settings is that the algorithm is not an entirely faithful abstraction of the network, i.e., it only partially captures true reasoning process of a model. We propose a solution where we combine different simple high-level models to produce a more faithful representation of the network. Through learning this combination, we can model neural networks as being in different computational states depending on the input provided, which we show is more accurate to GPT-2 small fine-tuned on two toy tasks. We observe a trade off between the strength of an interpretability hypothesis, which we define in terms of the number of inputs explained by the high-level models, and its faithfulness, which we define as the interchange intervention accuracy. Our method allows us to modulate between the two, providing the most accurate combination of models that describe the behavior of a neural network given a faithfulness level.
BibTeX
@inproceedings{pislar2025combining,
  author = {Theodora-Mara Pîslar and Sara Magliacane and Atticus Geiger},
  title = {{Combining Causal Models for More Accurate Abstractions of Neural Networks}},
  booktitle = {CLeaR},
  year = {2025},
  url = {https://openreview.net/forum?id=mVftlEi1CD},
}

Loading related content...

Figure