Combining Causal Models for More Accurate Abstractions of Neural Networks

Theodora-Mara Pîslar, Sara Magliacane, Atticus Geiger

Abstract

Mechanistic interpretability aims to reverse engineer neural networks by uncovering which high-level algorithms they implement. Causal abstraction provides a precise notion of when a network implements an algorithm, i.e., a causal model of the network contains low-level features that realize the high-level variables in a causal model of the algorithm. A typical problem in practical settings is that the algorithm is not an entirely faithful abstraction of the network, meaning it only partially captures the true reasoning process of a model. We propose a solution where we combine different simple high-level models to produce a more faithful representation of the network. Through learning this combination, we can model neural networks as being in different computational states depending on the input provided, which we show is more accurate to GPT 2-small fine-tuned on two toy tasks. We observe a trade-off between the strength of an interpretability hypothesis, which we define in terms of the number of inputs explained by the high-level models, and its faithfulness, which we define as the interchange intervention accuracy. Our method allows us to modulate between the two, providing the most accurate combination of models that describe the behavior of a neural network given a faithfulness level.

Introduction

Mechanistic interpretability seeks to "reverse engineer" neural networks by identifying human-understandable algorithms within their complex parameter spaces. A prominent framework for this is causal abstraction, which evaluates whether a high-level algorithm (e.g., "add $X$ and $Y$, then add $Z$") accurately describes the low-level activations of a model like GPT-2.

However, neural networks rarely implement a single algorithm in its purest form. A model might employ different strategies based on input characteristics or exist in transitional states between computational steps. Standard interpretability methods often attempt to impose a single, static causal model, leading to "unfaithful" explanations that only apply to a subset of inputs. This research introduces a method to "stitch together" multiple causal models, creating a more faithful representation of a network's internal logic by allowing it to be modeled as using different computational strategies for different inputs.

Methodology

The core innovation is the "combine" operation for causal models. Instead of selecting a single algorithm, the authors define a Combined Causal Model that functions as a piecewise mapping: it partitions the input space and assigns distinct sub-algorithms to different regions.

The approach utilizes several key components:

  • Distributed Alignment Search (DAS): This technique identifies the specific linear subspaces within the neural network that correspond to variables in the high-level algorithm.
  • Evaluation Graphs: Faithfulness is represented as a graph where nodes are inputs and edges denote the success of an interchange intervention (swapping activations between inputs).
  • Greedy Partitioning: A heuristic algorithm clusters inputs into the largest possible groups that satisfy a predefined faithfulness threshold ($\lambda$). Inputs that cannot be explained by a structured algorithm are assigned to a "trivial" (black-box) model.
  • Faithfulness-Strength Trade-off: The researchers define Strength as the proportion of inputs explained by a non-trivial model. The objective is to maximize Strength while maintaining high Interchange Intervention Accuracy (IIA).

Experimental Setup

The method was evaluated on a fine-tuned GPT-2 small model across two controlled tasks:

  1. Arithmetic Task: Summing three integers ($X+Y+Z$) with 1,000 total input combinations.
  2. Boolean Logic Task: Evaluating complex expressions (e.g., NOT (NOT X AND Y)) with 64 possible input combinations.

These tasks provide a "ground truth" to verify if the combined models accurately capture the underlying logic.

Results and Discussion

The experiments revealed that neural networks are effectively piecewise algorithmic. A network's computational state is highly dependent on the input. In the arithmetic task, GPT-2 does not simply perform a single addition; it transitions through various partial summation states across its layers.

Interchange Intervention Accuracy (IIA) across layers for various causal model hypotheses in the arithmetic task.

As shown in the figure above, the method successfully maps the "computational journey" of values through transformer blocks. In layers 1-4, the model maintains separate representations of variables (evidenced by high IIA for individual variable models like $M^X, M^Y, M^Z$). From layers 5-11, there is a gradual transition where these individual models lose faithfulness, giving way to partial and eventually full sums ($M^{XYZ}$).

Key findings include:

  • Higher Interpretability: Combined models consistently provided stronger hypotheses than single models. In the Boolean task, combined models explained 75% to 88% of the data at 100% faithfulness, whereas individual models explained less than 50%.
  • Asymmetry in Attention: Early layers were more effective at representing $X+Y$ than $Y+Z$, likely due to the left-to-right causal attention mechanism of GPT-2.
  • The Trivial Model Escape Hatch: By incorporating a "trivial model" for unexplained inputs, the researchers can quantify the remaining "black box" portion of the network at any given faithfulness level.

Conclusion and Future Work

This work demonstrates a fundamental trade-off between the strength and faithfulness of interpretability hypotheses. By combining models, researchers can find the "sweet spot"—the most descriptive hypothesis that remains highly accurate.

Current limitations include scalability, as the greedy partitioning heuristic was tested on relatively small input spaces. Future work will explore more efficient partitioning strategies for real-world LLM behaviors, such as factual recall. Additionally, integrating this causal approach with Sparse Autoencoders (SAEs)—the current standard for feature discovery—could yield even more robust mechanistic explanations.

Summary updated

Abstract

Mechanistic interpretability aims to reverse engineer neural networks by uncovering which high-level algorithms they implement. Causal abstraction provides a precise notion of when a network implements an algorithm, i.e., a causal model of the network contains low-level features that realize the high-level variables in a causal model of the algorithm. A typical problem in practical settings is that the algorithm is not an entirely faithful abstraction of the network, meaning it only partially captures the true reasoning process of a model. We propose a solution where we combine different simple high-level models to produce a more faithful representation of the network. Through learning this combination, we can model neural networks as being in different computational states depending on the input provided, which we show is more accurate to GPT 2-small fine-tuned on two toy tasks. We observe a trade-off between the strength of an interpretability hypothesis, which we define in terms of the number of inputs explained by the high-level models, and its faithfulness, which we define as the interchange intervention accuracy. Our method allows us to modulate between the two, providing the most accurate combination of models that describe the behavior of a neural network given a faithfulness level.
BibTeX
@inproceedings{pislar2025combining,
  author = {Theodora-Mara Pîslar and Sara Magliacane and Atticus Geiger},
  title = {{Combining Causal Models for More Accurate Abstractions of Neural Networks}},
  booktitle = {CLeaR},
  year = {2025},
  url = {https://openreview.net/forum?id=mVftlEi1CD},
}
Figure