When a Transformer model makes a prediction, it generates an internal matrix of attention weights. Because these weights assign a fractional "importance" score to every input token or image region, it is incredibly tempting to visualize them as heatmaps and declare them a built-in explanation of the model's internal logic. If a model assigns a high attention weight to a specific word, the intuitive leap is to assume that word caused the output.
However, a major thread of research reveals that this intuition is frequently flawed. Understanding why attention weights can mislead us—and how they can be properly grounded—requires looking closely at the mathematical mechanics of the attention layer and how it aligns with human cognition.
The Mathematical Disconnect
To understand why raw attention weights fail as faithful explanations, we must look at what happens immediately after the attention matrix is computed. The attention weights are not the final output; they are coefficients used to compute a weighted sum of value vectors. If a token has a massive attention weight but its corresponding value vector has a near-zero norm, that token will have virtually no impact on the subsequent layer.
As Gianluigi Lopardo et al. demonstrate in Attention Meets Post-hoc Interpretability: A Mathematical Perspective, raw attention weights represent only an isolated snapshot of a highly non-linear pipeline. They completely ignore:
- The magnitudes of the value vectors being mixed.
- The transformations applied by subsequent feed-forward layers.
- The negative contributions or cancellations that occur during vector addition.
By deriving explicit mathematical formulations for established post-hoc explanation methods like LIME and gradient-based attributions, the authors prove that true feature importance often diverges significantly from raw attention weights.
Even when specific attention heads appear to isolate semantic concepts perfectly—such as the spontaneous emergence of a "laughing head" that flags punchlines in Laughing Heads: Can Transformers Detect What Makes a Sentence Funny? by Maxime Peyrard et al.—it remains ambiguous whether these weights drive the decision causally or act as incidental markers of salient tokens.
Disconnect from Human Cognition
Another way to evaluate attention as an explanation is to test whether it aligns with how humans process information. If attention maps reflect genuine reasoning, they should correlate with human cognitive signals, such as eye-tracking data.
In Do Transformer Models Show Similar Attention Patterns to Task-Specific Human Gaze?, Oliver Eberle et al. show that raw attention distributions correlate poorly with human gaze. However, the authors find that if you calculate "attention flow"—a metric that tracks how information propagates and mixes across all layers of the network rather than looking at a single layer's weights—the alignment with human gaze improves significantly.
This same disconnect extends to computer vision. In Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?, Abhishek Das et al. demonstrated a stark divergence between machine attention maps and human visual focus when solving visual reasoning tasks.
Because raw attention weights do not reliably map to actual reasoning, presenting them to users as a form of transparency can be actively dangerous. Andres Carvallo De Ferari argues in User Perception of Attention Visualizations: Effects on Interpretability Across Evidence-Based Medical Documents that providing raw attention visualizations to domain experts in high-stakes fields like medicine can be misleading, creating a false sense of trust in a model's output based on non-causal correlations.
Grounding Attention Mechanistically
If raw attention weights are unreliable heuristics, how can we make sense of why they work so well in practice? Researchers are bridging this gap by grounding the attention mechanism in formal mathematical frameworks.
Instead of treating attention as an arbitrary weighting trick, Trenton Bricken et al. reveal in Attention Approximates Sparse Distributed Memory that the attention operation is mathematically equivalent to Kanerva's Sparse Distributed Memory (SDM), a classic biological model of associative memory. By showing that the softmax operation in attention approximates the intersection of high-dimensional hyperspheres, they provide a geometric explanation: attention works because it acts as an associative memory retrieval system, mapping queries to stored patterns.
Furthermore, understanding how these weights evolve during training helps explain their final state. In On the Dynamics of Training Attention Models, Haoye Lu et al. mathematically prove that a "mutual promotion" synergy occurs during optimization: a token's attention score and its embedding norm amplify one another. This feedback loop forces the model to naturally converge on the most discriminative features, explaining the mathematical trajectory behind feature selection.
Bridging Explanation and Performance
If natural training dynamics create unfaithful attention maps, can we fix both interpretability and performance by forcing attention to be "correct"?
Chenxi Liu et al. investigated this in Attention Correctness in Neural Image Captioning by explicitly supervising the attention layer during training, forcing the model to look at human-annotated regions. They discovered that aligning machine attention with human focus does not just make the internal maps more interpretable—it directly improves the quality of the generated captions.
This suggests that while raw, unsupervised attention weights should not be blindly trusted as post-hoc explanations, intentionally structuring or supervising the attention mechanism can bridge the gap, turning attention maps into both a tool for better performance and a faithful window into model reasoning.