Tracking Information Flow Across Transformer Layers

28 concepts26 papers1 proposal

When trying to understand why a Transformer model made a specific prediction, a natural first instinct is to look at its self-attention weights. If layer 12 puts a high weight on a specific token, it seems intuitive that this token must be highly important.

However, this interpretation is fundamentally deceptive. To understand why, imagine a game of "telephone" played across the layers of a deep network. In the first layer, Token A might attend to Token B. In the second layer, Token C attends to Token A. By the time the data reaches the final layer, a high attention weight directed at Token C does not mean the model is focusing on the original input token C; instead, it is focusing on a representation that has already swallowed up information from Tokens A, B, and many others.

As first formalized by Samira Abnar et al. in Quantifying Attention Flow in Transformers, information becomes increasingly mixed and diluted as it propagates through a deep network. To accurately attribute importance to the original input features, we cannot look at any single layer's attention map in isolation. Instead, we must track and aggregate how information flows across the entire multi-layer hierarchy.

The Mechanics of Attention Rollout

To untangle this layer-by-layer mixing, we can model the Transformer as a directed graph where the nodes are tokens at specific layers and the edges represent the attention weights between them. If we assume that self-attention forms a linear combination of token representations at each step, we can calculate the cumulative influence of the input layer on any deep layer by recursively multiplying the attention matrices of successive layers. This technique is known as Attention Rollout.

However, a pure multiplication of raw attention matrices assumes that information only moves through the attention mechanism. In reality, Transformers rely heavily on residual connections:

$$x_{l+1} = x_l + \text{Attention}(x_l)$$

Without accounting for this identity path, a rollout map would assume tokens completely lose their own identity at each step. To fix this, attention rollout adds an identity matrix $I$ to the attention matrix $A$ at each layer, weighting them to reflect how much a token maintains its own state versus absorbing context from its neighbors:

$$\tilde{A}_l = 0.5 A_l + 0.5 I$$

By recursively multiplying these adjusted matrices across all layers, we map high-level, abstract representations back to the original input tokens. This global aggregation provides a much more stable and faithful view of feature relevance than single-layer snapshots (Michael Neely et al., A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing).

Beyond Linear Multiplication: Flow and Full-Block Interactions

While attention rollout provides an elegant geometric intuition, treating the network as a simple chain of matrix multiplications has limitations. For instance, it can overshoot the actual information capacity of the network. To address this, Samira Abnar et al. proposed Attention Flow, which treats the Transformer architecture as a literal flow network and uses maximum flow algorithms to calculate token influence. This method yields attributions that closely mirror cognitive baselines, such as task-specific human eye-tracking data, as demonstrated by the authors of Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?.

Other approaches point out that attention weights alone ignore the transformations applied by layer normalization and feed-forward networks (MLPs). To capture these non-linearities, the authors of Measuring the Mixing of Contextual Information in the Transformer developed ALTI (Aggregation of Layer-wise Token-to-token Interactions). ALTI uses the $\ell_1$ norm of the full block's mathematical contributions to build a more faithful contribution graph, tracking exactly how much an original input token influences deep representations.

Alternatively, researchers have combined attention metrics with gradient-based saliency maps. For example, Sahana Ramnath et al. utilized layer-wise attribution via Integrated Gradients to track how models shift focus from question words to contextual answers across depths (Towards Interpreting BERT for Reading Comprehension Based QA), while Yuhao Wang et al. combined attention and gradients to map how Large Language Models reconcile internal memory with external context across layers (Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation).

Why Cross-Layer Tracking Works: Empirical Evidence

Tracking information flow across layers unlocks significant capabilities across modalities by resolving several fundamental issues with deep representations:

Alternative Paradigms of Cross-Layer Aggregation

Not all multi-layer approaches rely on recursive multiplication. Some architectures explicitly learn to aggregate or manipulate information across depths to optimize performance or interpretability:

Key Foundational Work

Extracting Syntactic Trees from Transformer Encoder Self-Attentions (2018) — contributed 1 concept elements to this direction.
Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attention_rollout_deea8f17,
  title = {attention rollout},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-rollout/rdr_deea8f1748b5407cb666eaf9c519c60e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_attention_rollout_deea8f17_v1,
  title = {attention rollout},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-rollout/rdr_deea8f1748b5407cb666eaf9c519c60e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake