When trying to understand why a Transformer model made a specific prediction, a natural first instinct is to look at its self-attention weights. If layer 12 puts a high weight on a specific token, it seems intuitive that this token must be highly important.
However, this interpretation is fundamentally deceptive. To understand why, imagine a game of "telephone" played across the layers of a deep network. In the first layer, Token A might attend to Token B. In the second layer, Token C attends to Token A. By the time the data reaches the final layer, a high attention weight directed at Token C does not mean the model is focusing on the original input token C; instead, it is focusing on a representation that has already swallowed up information from Tokens A, B, and many others.
As first formalized by Samira Abnar et al. in Quantifying Attention Flow in Transformers, information becomes increasingly mixed and diluted as it propagates through a deep network. To accurately attribute importance to the original input features, we cannot look at any single layer's attention map in isolation. Instead, we must track and aggregate how information flows across the entire multi-layer hierarchy.
The Mechanics of Attention Rollout
To untangle this layer-by-layer mixing, we can model the Transformer as a directed graph where the nodes are tokens at specific layers and the edges represent the attention weights between them. If we assume that self-attention forms a linear combination of token representations at each step, we can calculate the cumulative influence of the input layer on any deep layer by recursively multiplying the attention matrices of successive layers. This technique is known as Attention Rollout.
However, a pure multiplication of raw attention matrices assumes that information only moves through the attention mechanism. In reality, Transformers rely heavily on residual connections:
$$x_{l+1} = x_l + \text{Attention}(x_l)$$
Without accounting for this identity path, a rollout map would assume tokens completely lose their own identity at each step. To fix this, attention rollout adds an identity matrix $I$ to the attention matrix $A$ at each layer, weighting them to reflect how much a token maintains its own state versus absorbing context from its neighbors:
$$\tilde{A}_l = 0.5 A_l + 0.5 I$$
By recursively multiplying these adjusted matrices across all layers, we map high-level, abstract representations back to the original input tokens. This global aggregation provides a much more stable and faithful view of feature relevance than single-layer snapshots (Michael Neely et al., A Song of (Dis)agreement: Evaluating the Evaluation of Explainable Artificial Intelligence in Natural Language Processing).
Beyond Linear Multiplication: Flow and Full-Block Interactions
While attention rollout provides an elegant geometric intuition, treating the network as a simple chain of matrix multiplications has limitations. For instance, it can overshoot the actual information capacity of the network. To address this, Samira Abnar et al. proposed Attention Flow, which treats the Transformer architecture as a literal flow network and uses maximum flow algorithms to calculate token influence. This method yields attributions that closely mirror cognitive baselines, such as task-specific human eye-tracking data, as demonstrated by the authors of Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?.
Other approaches point out that attention weights alone ignore the transformations applied by layer normalization and feed-forward networks (MLPs). To capture these non-linearities, the authors of Measuring the Mixing of Contextual Information in the Transformer developed ALTI (Aggregation of Layer-wise Token-to-token Interactions). ALTI uses the $\ell_1$ norm of the full block's mathematical contributions to build a more faithful contribution graph, tracking exactly how much an original input token influences deep representations.
Alternatively, researchers have combined attention metrics with gradient-based saliency maps. For example, Sahana Ramnath et al. utilized layer-wise attribution via Integrated Gradients to track how models shift focus from question words to contextual answers across depths (Towards Interpreting BERT for Reading Comprehension Based QA), while Yuhao Wang et al. combined attention and gradients to map how Large Language Models reconcile internal memory with external context across layers (Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation).
Why Cross-Layer Tracking Works: Empirical Evidence
Tracking information flow across layers unlocks significant capabilities across modalities by resolving several fundamental issues with deep representations:
- Preserving Vanishing Details: In vision transformers, highly discriminative details (like the specific shape of a bird's beak) often become diffused in the final layers when viewed in isolation. Ju He et al. demonstrated in TransFG: A Transformer Architecture for Fine-Grained Recognition that recursively multiplying attention matrices across the hierarchy allows fine-grained classification models to accurately pinpoint highly localized, salient image patches that final-layer activations obscure.
- Zero-Shot Structure Discovery: Because rollout captures intrinsic semantic relationships, it can reveal coherent structures without any task-specific training. Anastasios Panagopoulos et al. utilized attention rollout in Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies to extract clean, object-centric scene segmentations directly from generic self-supervised models like DINOv2. Similarly, Samuel Black et al. leveraged rollout to map high-level image similarity back to exact input pixels, revealing that transformers naturally focus on global structures rather than local textures (Visualizing Paired Image Similarity in Transformer Networks).
- Stability and Faithfulness: Post-hoc explanation methods like gradient propagation can be highly unstable and computationally expensive. In multimodal remote sensing, Hiba Najjar demonstrated that intrinsic mechanisms like attention rollout provide stable, agronomically meaningful temporal explanations for crop yield predictions without relying on noisy gradients (Intrinsic Explainability of Multimodal Learning for Crop Yield Prediction; Intrinsic explainability of multimodal learning for crop yield simulation by Hiba Najjar et al.).
Alternative Paradigms of Cross-Layer Aggregation
Not all multi-layer approaches rely on recursive multiplication. Some architectures explicitly learn to aggregate or manipulate information across depths to optimize performance or interpretability:
- Learned Aggregation: Instead of using only the final layer, Masahiro Kaneko et al. introduced Multi-Head Multi-Layer Attention (MHMLA) to dynamically aggregate intermediate representations across all layers, significantly improving grammatical error detection (Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection).
- Averaging and Sharpening: Rather than multiplying, Xiaotian Lu et al. showed that simply averaging self-attention weights across all layers provides highly stable and human-interpretable saliency maps (Evaluating Saliency Explanations in NLP by Crowdsourcing). In contrast, for long-context multi-document QA, Jiakai Li et al. monitored cross-layer attention aggregation to dynamically sharpen attention on relevant context while suppressing noise (DSAS: A Universal Plug-and-Play Framework for Attention Optimization in Multi-Document Question Answering).
- Supervising the Flow: Rather than treating cross-layer tracking purely as an interpretability tool, Jules Samaran et al. used attention rollout to directly supervise the integrated path of information through a network, proving that supervising the cumulative flow provides a much more robust grounding signal than supervising individual attention heads (Attending Self-Attention: A Case Study of Visually Grounded Supervision in Vision-and-Language Transformers).
Key Foundational Work
Extracting Syntactic Trees from Transformer Encoder Self-Attentions (2018) — contributed 1 concept elements to this direction.
Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection (2019) — contributed 1 concept elements to this direction.