Abstract
Transformer-based language models have achieved significant success; however, their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. While previous studies have demonstrated that these models implicitly embed reasoning trees, humans typically employ various distinct logical reasoning mechanisms to complete the same task. It is still unclear which multi-step reasoning mechanisms are used by language models to solve such tasks. In this paper, we aim to address this question by investigating the mechanistic interpretability of language models, particularly in the context of multi-step reasoning tasks. Specifically, we employ circuit analysis and self-influence functions to evaluate the changing importance of each token throughout the reasoning process, allowing us to map the reasoning paths adopted by the model. We apply this methodology to the GPT-2 model on a prediction task (IOI) and demonstrate that the underlying circuits reveal a human-interpretable reasoning process used by the model.
Introduction
Despite the ubiquity of Transformer-based architectures in modern artificial intelligence, their internal decision-making processes remain a "black box." While we observe these models performing complex multi-step reasoning, the specific internal paths taken to reach a conclusion are often obscured by the model's high dimensionality. Existing research in Mechanistic Interpretability (MI) typically bifurcates into two categories: identifying "circuits" (the sub-graphs of neurons responsible for specific tasks) or utilizing "influence functions" to trace the impact of training data. However, neither approach fully captures the dynamic, layer-by-layer evolution of how a model weighs input tokens during a specific inference. This paper introduces the Self-Influence Circuit Analysis Framework (SICAF) to bridge this gap, tracing token importance through the model's internal architecture to reveal interpretable reasoning paths.
Methodology: The SICAF Framework
The SICAF framework consists of a three-stage process designed to isolate and analyze the reasoning mechanisms of a language model.
1. Automatic Circuit Identification
To manage the complexity of the full model, SICAF first identifies a "circuit"—a minimal sub-graph responsible for the task at hand. The researchers utilize Edge Attribution Patching (EAP) and its advanced variants, EAP-IG and EAP-IG-KL (based on Kullback-Leibler divergence). These methods allow for the identification of a circuit containing only 1-2% of the model's total edges while maintaining the majority of the model's performance.
2. Layer-wise Self-Influence Calculation
Once the circuit is isolated, the framework calculates "self-influence" scores for each token at every layer. This metric measures the impact of a specific input token on the model's parameters at that specific stage of processing. To ensure computational feasibility for large-scale models, the authors employ a Hessian-Vector Product (HVP) approach combined with a recursive Taylor expansion. This avoids the memory-intensive requirements of traditional inverse Hessian matrix calculations.
3. Thought Process Inference
By tracking the evolution of these influence scores from the initial input (Layer 0) to the final output (Layer 11 in GPT-2 Small), the framework maps a "reasoning tree." This tree visualizes which entities or linguistic components the model prioritizes at each step of its internal computation.
Experimental Setup and Datasets
The framework was evaluated using the Indirect Object Identification (IOI) task. In this task, a model is provided with a prompt such as: "When Amy and Laura got a snack... Laura decided to give it to [blank]." The model must correctly predict "Amy."
This task is a benchmark for multi-step reasoning because it requires the model to:
- Identify the names (entities) in the sentence.
- Recognize which name is repeated (the subject).
- Deduce that the non-repeated name is the recipient (the indirect object).
The study utilized GPT-2 Small and compared performance on "clean" sentences against "corrupted" versions to validate the faithfulness of the identified circuits.
Key Insights and Results
Circuit Faithfulness and Efficiency
The research demonstrated that the identified circuits are highly efficient. A circuit consisting of approximately 800 edges was able to recover over 85% of the full model's performance on the IOI task. Notably, the EAP-IG-KL method proved superior to standard EAP, achieving faithfulness scores above 0.8.
The "U-Shaped" Importance Profile
Analysis of the layers revealed a "U-shaped" distribution of parameter importance. The model's key reasoning parameters are primarily concentrated in the first layer (initial feature extraction) and the final few layers (decision aggregation). The intermediate layers serve as a refinement stage where relational structures between tokens are established.
Human-Interpretable Logic
The reasoning trees extracted via SICAF align closely with human logic. In the early layers, the model focuses on identifying key entities (e.g., "Christina" or "Amy"). In the middle layers, the focus shifts toward the action verb (e.g., "gave"). Finally, the model synthesizes these elements to identify the correct recipient. This confirms that tokens like specific names and verbs act as the "anchors" of the model's reasoning process.
Discussion and Limitations
While SICAF provides a powerful lens into the Transformer's "thought process," several challenges remain. The current study focused on GPT-2 Small and the IOI task. It remains to be seen how these circuits scale in complexity within massive models like GPT-4 or Llama-3. Furthermore, while the IOI task is a clear example of linguistic reasoning, future work must explore whether SICAF can map reasoning in more abstract domains, such as mathematical logic or code generation. Finally, despite HVP optimizations, the computational cost of layer-wise influence analysis remains high, necessitating further optimization for real-time applications.
Summary updated