Transformer-specific Interpretability

Hosein Mohebbi, Jaap Jumelet, Michael Hanna, Afra Alishahi, Willem Zuidema2024 · DOI

Abstract: Hosein Mohebbi, Jaap Jumelet, Michael Hanna, Afra Alishahi, Willem Zuidema. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts.

2024.

Introduction

As Transformers have become the foundational architecture for modern artificial intelligence, the urgency to understand their internal decision-making processes has intensified. Historically, researchers relied on "model-agnostic" interpretability techniques—such as gradient-based saliency or occlusion—which treat the model as a black box. However, these methods often produce conflicting results and fail to account for the unique mathematical structure of the Transformer. This paper argues for a shift toward Transformer-specific interpretability, leveraging the model's inherent architecture—specifically its residual streams and attention heads—to provide more faithful explanations of how these models "think."

The Limitations of Model-Agnostic Methods

The primary challenge with traditional interpretability tools is the "disagreement problem." Different techniques frequently yield wildly different explanations for the same prediction, leaving researchers unsure which (if any) is correct. Furthermore, model-agnostic methods often ignore the additive nature of residual connections and the complex parallel interactions between tokens. By treating the Transformer as a generic function, these methods miss the nuances of how information is actually transformed across layers.

A Taxonomy of Transformer-Native Interpretability

The authors categorize modern, architecture-aware interpretability into two primary frameworks:

1. Context Mixing Analysis

These methods aim to quantify the movement of information between tokens. Rather than relying solely on raw attention scores—which can be misleading due to layer normalization and residual bypasses—techniques like Attention-Norm, ALTI, and GlobEnc measure the actual "flow" of information. By incorporating the effects of every component in the block, these methods provide a more accurate map of how a model integrates context from across a sequence.

2. Mechanistic Interpretability

This approach treats the Transformer like a computer program to be reverse-engineered. The goal is to identify "circuits"—specific subnetworks of neurons and attention heads that perform discrete logical tasks. Key tools in this domain include:

  • Path Patching: Causally intervening on specific mathematical paths to see how they affect the output.
  • Logit Lens: Projecting intermediate layer outputs directly into the final vocabulary space to see how the model's "prediction" evolves layer-by-layer.

The Residual Stream as a Communication Bus

A central insight of this research is the conceptualization of the residual stream as a shared communication bus. In this view, different layers do not simply transform the input; they "read" from and "write" to a continuous stream of information. This allows researchers to track how specific concepts—such as grammatical plurality or gender—are incrementally constructed. It shifts the focus from asking what the model looks at to how it computes, moving from simple saliency maps to complex functional circuits.

Key Findings and Landmark Results

While the paper serves as a tutorial roadmap, it synthesizes several landmark findings in the field:

  • Task Localization: Researchers have successfully identified "induction heads" (responsible for completing repetitive patterns) and specific "indirect object identification" circuits in models like GPT-2.
  • Improved Faithfulness: Transformer-specific methods like ALTI have proven more "faithful" to the model's actual decision-making process than general-purpose gradient methods, which are often noisy and unstable.
  • Cross-Domain Generalization: These architecture-specific tools remain effective across different modalities, including speech and music, suggesting that the Transformer's structure imposes a consistent logic regardless of the input data.

Challenges and Future Directions

Despite significant progress, several hurdles remain for the field of interpretability:

  • Scalability: While reverse-engineering is possible for smaller models (e.g., GPT-2 Small), applying these techniques to Large Language Models (LLMs) with hundreds of billions of parameters is analytically daunting.
  • Polysemanticity: Individual neurons often respond to multiple, unrelated concepts, making it difficult to isolate the "meaning" of specific parts of a circuit.
  • Stability: There is ongoing debate regarding how well these methods transfer between different model families or training regimes.

The authors conclude that the future of the field lies in automated circuit discovery and developing better tools to decompose the Feed-Forward Network (FFN) layers, which currently remain more opaque than the attention mechanism.

Summary updated

BibTeX
@misc{mohebbi2024transformer,
  author = {Hosein Mohebbi and Jaap Jumelet and Michael Hanna and Afra Alishahi and Willem Zuidema},
  title = {{Transformer-specific Interpretability}},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-tutorials.4},
}
Figure