Transformers are notoriously difficult to interpret because their internal representations change dynamically across dozens of layers. However, a structural property of the Transformer architecture offers a powerful foothold for mechanistic interpretability: the residual stream. Instead of completely replacing the hidden state at each layer, a Transformer operates by adding updates from attention heads and feed-forward networks (FFNs) to a running vector.
Because addition is linear, we can mathematically untangle this stream. The final output of a model—and even its final prediction logits—can be expressed as a direct, linear summation of contributions from every preceding component and input token. This approach allows us to bypass the "black box" view and trace exactly how information flows through the network.
The Core Intuition: Linearity in a Residual Network
To understand why this works, consider the standard forward pass of a Transformer layer. If $x_0$ is the initial token embedding, the hidden state at layer $l$ is defined recursively as:
$$x_l = x_{l-1} + \text{Attn}_l(x_{l-1}) + \text{FFN}_l(x_{l-1})$$
By unrolling this recursion all the way to the final layer $L$, the final representation $x_L$ is simply the sum of the initial embedding and the outputs of every single attention and FFN sub-layer:
$$x_L = x_0 + \sum_{l=1}^L \text{Attn}_l(x_{l-1}) + \sum_{l=1}^L \text{FFN}_l(x_{l-1})$$
As Javier Ferrando et al. demonstrate in Explaining How Transformers Use Context to Build Predictions, this additive structure means that the final output logit used to predict the next token is also a linear combination of these layer-wise updates. By multiplying each component's output vector directly with the final vocabulary embedding matrix (the unembedding layer), researchers can perform direct logit attribution.
As Jack Merullo et al. show in Circuit Component Reuse Across Tasks in Transformer Language Models, this technique allows us to isolate the exact attention heads responsible for shifting the probability of a specific target token. This linearity is not limited to language; Yossi Gandelsman et al. leverage it in Interpreting CLIP's Image Representation via Text-Based Decomposition to break down Vision Transformer (ViT) representations into individual attention head contributions, mapping specialized "expert" heads to specific visual properties like color or shape.
Tracking Token-Level Information Flow
While attributing outputs to specific layers or heads is useful, we often want to know which input tokens drove a decision. Traditional approaches rely heavily on raw attention maps, but Goro Kobayashi et al. argue in Incorporating Residual and Normalization Layers into Analysis of Masked Language Models that attention weights alone provide a misleading picture because they ignore the massive role of the residual connection. In fact, their decomposition reveals that Transformers are highly conservative, primarily preserving original token information through skip connections while mixing context at a much lower rate.
To accurately track token interactions, methods like ALTI (Aggregation of Layer-wise Token-to-token Interactions) and ALTI+, introduced by Javier Ferrando et al. in Measuring the Mixing of Contextual Information in the Transformer and Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer, decompose the entire attention block—including residual connections and layer normalization—into a weighted sum of transformed input vectors. This allows researchers to map exactly how much a model relies on source text versus previously generated target tokens, revealing that models rely heavily on internal linguistic patterns over source context during hallucinations.
Similarly, Byung-Doh Oh et al. use Token-wise Decomposition of Autoregressive Language Model Hidden States for Analyzing Model Predictions to show that autoregressive predictions are heavily driven by simple word collocations and verbatim repetitions. To avoid losing rich representational data during this process, Ali Modarressi et al. propose DecompX in DecompX: Explaining Transformers Decisions by Propagating Token Decomposition, which propagates full-dimensional token-specific contribution vectors through the network rather than reducing them prematurely to scalars. Even outside pure linear tracking, adaptations of Deep Taylor Decomposition by Daniel Hershcovich et al. demonstrate that resolving the complexities of skip connections yields highly reliable token attributions that significantly reduce human error rates in real-world tasks like fake news detection.
Overcoming the Challenge of Non-Linearities
If the residual stream is additive, why isn't interpretability trivial? The challenge lies in the non-linear operations interspersed between additions: Layer Normalization (LayerNorm) and activation functions (like ReLU or GELU) inside the FFNs. These operations break pure linearity, meaning a simple sum of inputs does not perfectly equal the output.
Researchers have developed clever mathematical workarounds to maintain the integrity of the decomposition:
- Freezing Activations: In DePass: Unified Feature Attributing by Simple Decomposed Forward Pass, Xiangyu Hong et al. introduce a framework that runs a standard forward pass, records the non-linear activation values, and then "freezes" them. Because the activations are fixed constants during the attribution phase, the remaining operations behave as purely linear transformations, allowing for perfectly faithful additive decomposition.
- Jet Expansions: Yihong Chen et al. tackle this in Decomposing LLM Computation with Jets by using "jet operators"—a generalization of Taylor series expansions. This framework unrolls the nested, entangled computations of an LLM into explicit, additive polynomial paths directly from the model's weights without requiring external datasets.
- Graph Traversal: In Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP, Sriram Balasubramanian et al. employ a recursive graph traversal algorithm to isolate the contribution vectors of attention heads and MLPs, mapping them into a shared text-image space to interpret what arbitrary vision models are seeing.
What Decomposition Reveals About Model Mechanics
By breaking the residual stream down into its constituent parts, researchers can audit, manipulate, and discover structural properties of models that are invisible from the outside.
Bias Mitigation
Hanzhang Zhou et al. introduce UniBias in UniBias: Unveiling and Mitigating LLM Bias through Internal Attention and FFN Manipulation. By decomposing the residual stream contributions of individual FFN vectors and attention heads, they identify and mask specific biased components during inference, improving model robustness and reducing sensitivity to prompt formatting without retraining.
Long-Range Routing in MoEs
In Mixture-of-Experts (MoE) architectures, Wengang Li et al. show in Understanding Cross-layer Contributions to Mixture-of-Experts Routing in LLMs that router inputs can be recursively decomposed. They discovered that routing decisions deep in a network are not purely local; instead, they are heavily driven by specific experts chosen in much earlier layers.
Feature Decoupling in Vision Transformers
In large Vision Transformers, component contribution analysis via $L_2$ norms reveals that global representations eventually decouple from local patch features. Alexander Lappe et al. demonstrate in Register and [CLS] tokens induce a decoupling of local and global features in large ViTs that as models scale, information flow shifts heavily toward internal skip connections and register tokens rather than direct patch integration, rendering traditional attention maps misleading.
Embedding Geometry vs. Function
Timothee Mickus et al. use linear decomposition in Why Bother with Geometry? On the Relevance of Linear Decompositions of Transformer Embeddings to isolate the functional contributions of sub-layers, finding that spatial geometry is often highly inconsistent across random seeds despite stable functional performance, suggesting geometry may be a model-specific idiosyncrasy.
Algorithmic Capacity
Taking architectural decomposition to its logical extreme, Deqing Fu in When Do Transformers Learn Heuristics for Graph Connectivity? explicitly alters the standard Transformer architecture to append layer outputs to the residual stream rather than summing them. This complete disentanglement reveals a strict capacity limit ($3^L$ for an $L$-layer model) governing whether a model learns a robust algorithm or a brittle shortcut, proving how structural modifications to the residual stream can expose the fundamental limits of neural computation.
Key Foundational Work
How to Dissect a Muppet: The Structure of Transformer Embedding Spaces (2022) — contributed 2 concept elements to this direction.
Measuring the Mixing of Contextual Information in the Transformer (2022) — contributed 1 concept elements to this direction.
Incorporating Residual and Normalization Layers into Analysis of Masked Language Models (2021) — contributed 1 concept elements to this direction.