Abstract: Transformer-based language models have achieved significant success; however, their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. While previous studies have demonstrated that these models implicitly embed reasoning trees, humans typically employ various distinct logical reasoning mechanisms to complete the same task. It is still unclear which multi-step reasoning mechanisms are used by language models to solve such tasks. In this paper, we aim to address this question by investigating the mechanistic interpretability of language models, particularly in the context of multi-step reasoning tasks. Specifically, we employ circuit analysis and self-influence functions to evaluate the changing importance of each token throughout the reasoning process, allowing us to map the reasoning paths adopted by the model. We apply this methodology to the GPT-2 model on a prediction task (IOI) and demonstrate that the underlying circuits reveal a human-interpretable reasoning process used by the model.
Introduction
Despite the massive success of Transformer-based models, their internal decision-making remains a "black box." While we observe their ability to perform multi-step reasoning, understanding how that logic is distributed across layers is a significant challenge. Existing methods like Influence Functions are often computationally prohibitive for modern LLMs, while standard Circuit Analysis identifies active components without necessarily explaining the narrative of the reasoning process. This paper introduces a framework to bridge this gap, visualizing the "reasoning tree" a model follows during inference.
Methodology: The SICAF Framework
The researchers propose the Self-Influence Circuit Analysis Framework (SICAF), which operates in three distinct phases:
- Automatic Circuit Identification: Using gradient-based methods—specifically Edge Attribution Patching (EAP) and its variants (EAP-IG and EAP-IG-KL)—the framework identifies a "circuit," a tiny subgraph comprising roughly 1-2% of the model's edges responsible for a specific task.
- Layer-wise Self-Influence Calculation: The framework calculates "self-influence" scores for each token across the layers of the identified circuit. To maintain computational feasibility, the authors utilize Hessian-vector products (HVP) and recursive Taylor expansions, avoiding the calculation of massive inverse Hessian matrices.
- Thought Process Inference: By tracking how the importance (influence) of specific tokens shifts from the initial layers to the final output, the model's logical path is reconstructed.
Experimental Setup
The study focuses on the Indirect Object Identification (IOI) task. In a sentence like "When Amy and Laura got a snack, Laura decided to give it to...", the model must predict the correct recipient ("Amy"). This requires identifying names, recognizing the repeated name (the distractor), and inferring the remaining name as the target. The authors tested the model's robustness using clean and corrupted inputs to verify the faithfulness of the identified circuits.
Analysis of Reasoning Paths
A key finding is that the model's internal logic mirrors human-interpretable steps. Importance is often "U-shaped," concentrated in the first layers for initial processing and the final layers for decision synthesis, with middle layers refining relational structures.
In the IOI task, early layers focus on identifying key entities like "Amy" and "Laura." Middle layers interpret the action verb (e.g., "gave"), and final layers synthesize these components to reach the logical conclusion. This progression aligns with how a human might decompose the same puzzle.
Performance and Faithfulness
The identified circuits are remarkably efficient. Containing only 1-2% of the model's total edges, these subgraphs recovered over 85% of the full model's performance on the IOI task.
As shown in the performance metrics, methods like EAP-IG and EAP-IG-KL reach high faithfulness much faster than standard EAP, stabilizing after including a small fraction of parameters. This suggests that the core reasoning logic is highly localized within the broader network architecture.
Conclusion and Future Work
While SICAF successfully reverse-engineers reasoning in GPT-2 small, several avenues for future research remain. It is yet to be determined how this framework scales to frontier models with hundreds of billions of parameters. Furthermore, while the IOI task provides a clear benchmark, applying SICAF to more diverse and complex natural language reasoning tasks will be essential to proving its universality. Despite these limitations, the ability to calculate influence on task-specific circuits makes previously infeasible mechanistic analysis practically manageable.
Summary updated