When we want to understand what a Transformer model is "thinking," our first instinct is often to plot an attention map. It is an intuitive visualization: a matrix of weights indicating exactly how much focus a query token places on a key token.
However, this visual clarity is largely an illusion. An attention map only captures the raw query-key interaction matrix ($QK^T$). In reality, a Transformer layer is a complex pipeline of operations. Treating raw attention weights as a complete map of information flow is like trying to understand a factory's entire supply chain by looking only at the manager's routing slip. It tells you where things ought to go, but ignores whether anything was shipped, what was added along the way, or what bypassed the station entirely.
To build true intuition for why raw attention maps mislead us—and how modern interpretability research fixes this—we must look at the components that standard attention visualizations leave out: the residual stream, the value matrix, and the feed-forward networks.
The Residual Stream: The Highway of Token Identity
The most significant flaw in raw attention analysis is the omission of skip connections. A Transformer layer does not simply replace a token's embedding with a mixed combination of other tokens; it computes an update and adds it to the original embedding via a residual connection.
Input Token ───┬───────────────────────────────(+)───► Output
│ ▲
└──► [LayerNorm] ──► [Attention] ┘
If the residual connection carries the bulk of the signal, the attention weights only describe a small fraction of the final representation. By ignoring these identity mappings, traditional explanation methods miss the primary pathway of information transfer, leading to an incomplete understanding of the model's decision-making process (Qiang et al. (2022)).
When researchers account for the entire block—including residual connections and layer normalization—the story changes completely. Abnar et al. (2020) demonstrated that as information moves through deep layers, raw attention weights become increasingly misleading because information from different tokens gets mixed and diluted.
In fact, when you mathematically factor in the residual stream, Transformers turn out to be remarkably "conservative." Kobayashi et al. (2021) showed that the network primarily preserves the original token information, with contextual mixing playing a much smaller role than raw attention maps suggest. Lu et al. (2021) confirmed this architectural dependency, revealing that models like BERT rely heavily on skip connections to form representations, frequently utilizing these unmapped shortcuts to bypass primary processing layers entirely.
The Value Matrix: The Content Gatekeeper
Even if we isolate the attention block itself, focusing solely on the query-key interaction provides an incomplete view. The softmax of $QK^T$ determines the routing coefficients, but the Value matrix ($V$) determines the actual content being propagated.
Imagine an attention weight between two tokens is exceptionally high. If the corresponding value vector has a magnitude near zero, or if it projects into a subspace that the rest of the network ignores, no meaningful information is transferred. As Park et al. (2019) and Xu et al. (2024) argue, excluding the value matrix from visualization frameworks severely limits our ability to trace how information is transformed across layers.
Furthermore, the output of the attention mechanism is passed through subsequent linear projection matrices. These learned transformations modify token representations in ways that raw attention weights cannot account for (Chen et al. (2023)). Attention tells you the where, but the value and projection matrices determine the what.
The Feed-Forward Network: The Forgotten Engine
After information is mixed by attention, it enters a Feed-Forward Network (FFN) or Multi-Layer Perceptron (MLP) block. Mechanistic interpretability research has historically prioritized attention mechanisms while treating FFNs as a black box, despite the fact that FFNs contain the vast majority of a Transformer's parameters (Kobayashi et al. (2024)).
FFNs are not passive components; they scale, refine, and transform feature embeddings (Walker et al. (2025)). By omitting them, standard interpretability frameworks fail to map the model's internal logic (Friedman et al. (2025)). Wu et al. (2024) note that FFNs play a critical role in refining features before they reach the final output; ignoring them creates an incomplete picture of data flow.
This component neglect is a widespread issue in sequence model interpretability:
- In Vision Transformers (ViTs), focusing exclusively on attention weights neglects the significant contributions of MLPs and normalization layers, failing to provide a comprehensive understanding of how a model arrives at a specific decision (Böhle et al. (2023)).
- In "attention-free" architectures like gated-linear RNNs (e.g., Mamba, RWKV), explainability methods often focus exclusively on the recurrent core while ignoring peripheral components like convolutional layers and gating branches (Zimerman (2025)).
- Conversely, interpretability methods that focus only on mapping influential neuron pathways through FFNs create a reverse bias, leaving the routing contributions of multi-head attention modules unmapped (Wang et al. (2025)).
Tracking Holistic Information Flow
To move past these biases, a new paradigm of interpretability treats the entire Transformer layer as a unified computational graph, tracking how token vectors actually change in length, direction, and magnitude from input to output.
Raw Attention Map ──► [Captures QK^T Only] ──► Misleading Attribution
Holistic Flow ──► [Captures MHSA + Residuals + Values + FFN + Norms] ──► Faithful Attribution
Instead of isolating attention matrices, methods like Attention Rollout and Attention Flow (Abnar et al. (2020)) formalize information propagation as a flow network, tracking embeddings through both attention heads and residual paths. To make this computationally efficient and precise, frameworks like GlobEnc (Modarressi et al. (2022)) incorporate layer normalization and residual streams directly into token attribution, capturing exactly how a model preserves or modifies token identity across layers.
By shifting the focus from isolated weights to geometric token transformations across both attention and FFN blocks, modern frameworks like TokenTM (Wu et al. (2024)) eliminate the noise inherent to raw attention maps. The result is an explanation that reflects the true internal mechanics of the model, proving that in a Transformer, the whole architecture matters far more than any single component.
Key Foundational Work
Quantifying Attention Flow in Transformers (2020) — contributed 1 concept elements to this direction.
Attention is Not Only a Weight: Analyzing Transformers with Vector Norms (2020) — contributed 1 concept elements to this direction.
SANVis: Visual Analytics for Understanding Self-Attention Networks (2019) — contributed 1 concept elements to this direction.