The Decoupling of Attention Weights and Feature Importance

112 concepts83 papers2 proposals

In the visual language of neural networks, the "attention heatmap" has become the standard tool for transparency. When a model predicts a sentiment or answers a question, researchers often point to high attention weights as proof of what the model "looked at." However, a growing body of evidence suggests that these weights are frequently unreliable proxies for feature importance. The magnitude of an attention score does not necessarily dictate the functional impact of a token on the final output.

The Magnitude Fallacy

The most intuitive reason attention weights fail as explanations is that they only represent one half of a multiplication. In a Transformer, the output of an attention head is a weighted sum of transformed vectors. As Kobayashi et al. argue, judging a token's importance solely by its attention weight is like judging a song's volume by the amplifier settings while ignoring the volume of the instruments themselves. If a token is assigned a high attention weight but the underlying vector has a very small norm, its actual contribution to the residual stream is negligible.

This "norm-based" perspective reveals that models often use attention weights to filter out noise rather than highlight signal. Modarressi et al. further note that raw weights ignore residual connections, which allow significant information to bypass the attention mechanism entirely. Consequently, a token might be critical to a prediction despite having a low attention weight, or vice versa.

The Downstream Nullification

Even when an attention head successfully moves high-magnitude information, there is no guarantee that the rest of the network will use it. Lopardo et al. provide a mathematical perspective showing that attention weights fail to account for the final classification layer. This layer can effectively nullify or even invert the contribution of specific attention heads.

This leads to what Li et al. call "faithfulness violations," where features identified as "important" by attention may actually be working against the model's final decision. Because attention mechanisms do not distinguish between features that support a prediction and those that suppress it, a high weight only indicates that a feature was processed, not that it was beneficial.

Adversarial Attention and Combinatorial Shortcuts

If attention weights were a faithful explanation, we would expect a unique distribution of weights to correspond to a specific prediction. However, Jain et al. demonstrated the existence of "adversarial" attention distributions—alternative weightings that highlight completely different words but produce the exact same output. This suggests that the relationship between attention and prediction is not one-to-one, but many-to-one.

Furthermore, Bai et al. identify "combinatorial shortcuts," where downstream layers learn to exploit the pattern of attention weights as a form of encoded label information, rather than the actual content of the attended tokens. In these cases, the attention mechanism acts as a hidden signaling layer rather than a transparency tool.

Toward More Faithful Attribution

Because of these disconnects, researchers have turned to post-hoc attribution methods that measure causal influence rather than internal weights. Serrano et al. and Sydorova et al. show that gradient-based rankings and input perturbations are significantly more effective at identifying the features that actually drive model behavior.

These methods, such as the Mask Perturbation Model proposed by Zhang et al., focus on "indispensability"—if removing a token changes the output, that token is important. By calibrating attention to align with these causal metrics, researchers can move closer to models that are not just accurate, but truly interpretable.

Key Foundational Work

Is Attention Interpretable? (2019) — contributed 3 concept elements to this direction.
An Analysis of Attention over Clinical Notes for Predictive Tasks (2019) — contributed 3 concept elements to this direction.
An Analysis of Attention over Clinical Notes for Predictive Tasks (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attention_weight_unreliability_1984c077,
  title = {attention weight unreliability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-weight-unreliability/rdr_1984c077a429460aa9ac4a7ad907a2ed},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_attention_weight_unreliability_1984c077_v1,
  title = {attention weight unreliability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-weight-unreliability/rdr_1984c077a429460aa9ac4a7ad907a2ed/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake