Abstract
A faithful and interpretable explanation of an AI model's behavior and internal structure is a high-level explanation that is human-intelligible but also consistent with the known, but often opaque low-level causal details of the model. We argue that the theory of causal abstraction provides the mathematical foundations for the desired kinds of model explanations. In causal abstraction analysis, we use interventions on model-internal states to rigorously assess whether an interpretable high-level causal model is a faithful description of an AI model. Our contributions in this area are: (1) We generalize causal abstraction to cyclic causal structures and typed high-level variables. (2) We show how multi-source interchange interventions can be used to conduct causal abstraction analyses. (3) We define a notion of approximate causal abstraction that allows us to assess the degree to which a high-level causal model is a causal abstraction of a lower-level one. (4) We prove constructive causal abstraction can be decomposed into three operations we refer to as marginalization, variable-merge, and value-merge. (5) We formalize the XAI methods of LIME, causal effect estimation, causal mediation analysis, iterated nullspace projection, and circuit-based explanations as special cases of causal abstraction analysis.
Introduction: The Science of Reverse-Engineering
Deep learning models are notoriously "black boxes." While we understand their low-level mechanics—matrix multiplications and non-linearities—we often lack an understanding of the high-level "algorithms" they implement. The subfield of Mechanistic Interpretability (MI) aims to bridge this gap by finding human-intelligible simplifications of these models.
However, the field has historically lacked a common formal language. Researchers use various techniques (patching, scrubbing, steering, etc.) without a clear way to compare them or prove that a proposed high-level explanation is truly "faithful" to the model's actual internal logic. This paper, published in the Journal of Machine Learning Research, provides a rigorous mathematical framework for MI by unifying these disparate techniques under the banner of Causal Abstraction.
Theoretical Framework: Interventionals and Algebras
The paper extends the theory of causal abstraction through several novel formalisms designed to capture the complex, distributed representations found in modern AI:
- Interventionals: A generalization of standard interventions. Instead of just fixing a variable to a specific value, an "interventional" is a functional that transforms an old mechanism into a new one. This allows for much more flexible edits to a model's internal logic.
- Intervention Algebras: The authors identify a specific class of interventionals that behave like "hard interventions" in a different variable space. These algebras must satisfy properties of commutativity (order doesn't matter for different variables) and left-annihilativity (repeatedly intervening on the same variable only keeps the last effect).
- Constructive Causal Abstraction: A "lossy" but exact transformation that merges many low-level "microvariables" (neurons) into a few high-level "macrovariables" (concepts) through marginalization and merging.
Unifying Interpretability Methods
A major insight of this work is showing that nearly all major MI methods are mathematically equivalent to finding a causal abstraction. For instance, Sparse Autoencoders (SAEs) are viewed as learning a bijective translation into a sparse feature space where features are modular.
The authors provide a precise definition for "modular features": features are modular if they form an intervention algebra accessed via a bijective translation. This moves the debate beyond "neurons vs. directions" to a more general algebraic view, supporting non-linear abstractions of model internals.
Key Results and Faithfulness
The paper establishes several foundational theorems for the field:
- Representation Theorem: The authors prove that any set of operations satisfying the properties of an intervention algebra is isomorphic to a set of hard interventions on some (potentially hidden) variable space.
- Decomposition of Alignments: They prove that any complex alignment between a low-level and high-level model can be decomposed into three fundamental operations: marginalization, variable merging, and value merging.
- Graded Faithfulness: Faithfulness isn't all-or-nothing. The paper introduces a metric for "approximate transformation," allowing researchers to quantify how well a high-level explanation matches the model's behavior across a distribution of inputs and interventions.
Limitations and Future Work
While the framework is a significant step toward a formal science of MI, challenges remain. The current framework is primarily built for deterministic models. Furthermore, while the framework is excellent for verifying if a proposed algorithm is a faithful abstraction, it doesn't solve the "search problem" of discovering that algorithm from scratch for a trillion-parameter model. Finally, calculating exact causal abstractions for massive, real-world LLMs remains computationally daunting, suggesting a need for more efficient approximation algorithms.
Summary updated