Why Different Attention Patterns Produce Identical Outputs

17 concepts8 papers1 proposal

When we visualize a Transformer's attention map, we tend to treat it as a transparent window into the model's decision-making process. If a sentiment classifier highlights the word "excellent," we naturally assume that this specific token drove the prediction.

However, a growing body of research reveals a fundamental mathematical hurdle to this interpretation: non-identifiability. In many settings, the relationship between a model's internal components—whether they are attention weights, hidden parameters, or post-hoc saliency maps—and its final outputs is many-to-one. This means that infinitely many distinct internal configurations can yield the exact same behavior, rendering a single visual explanation inherently non-unique.

The Geometry of the Attention Null Space

To understand why attention maps can be misleading, we must look at the linear algebra underlying the attention mechanism. An attention head takes a sequence of input tokens, computes a distribution of attention weights, and uses these weights to compute a weighted average of the token "Value" representations.

As Brunner et al. demonstrate, a critical dimensionality mismatch occurs when the input sequence length ($n$) exceeds the internal token dimensionality ($d$) of the attention head. Because the head projects a higher-dimensional space of attention weights down to a lower-dimensional vector space, the underlying mapping becomes non-injective.

This mismatch creates a mathematical null space. You can alter the attention weights along the vectors in this null space, completely changing which tokens are visually highlighted in a heatmap, without changing the resulting output representation by even a fraction. Consequently, raw attention maps are fundamentally non-identifiable; they represent just one arbitrary choice among an infinite family of alternative distributions that compute the exact same output.

The Empirical Debate: Is Attention Explanation?

This geometric reality sparked a major debate regarding the validity of attention as an explanatory tool. Jain et al. provided empirical weight to this problem by systematically constructing "adversarial" attention distributions. They showed that for many NLP tasks, one can easily find alternative attention maps that highlight entirely different sets of features but produce identical model outputs, concluding that attention cannot be treated as a faithful explanation.

[Input Tokens] ---> [Infinitely Many Attention Maps (Null Space)] ---> [Single Identical Output]

However, the requirement for absolute uniqueness might be an overly restrictive standard for faithfulness. Wiegreffe et al. countered this view, arguing that the existence of alternative paths does not automatically invalidate the path the model actually took. Even if multiple valid attention distributions could yield the same prediction, the specific distribution selected during the model's forward pass still reflects its actual execution trace.

To navigate this impasse, Bibal et al. synthesized the debate, noting that while alternative distributions are mathematically possible, the core challenge is that non-uniqueness prevents us from proving a causal link between a specific attention pattern and the final prediction. To address this, researchers have begun designing architectures to explicitly bypass this ambiguity. For instance, Bhardwaj et al. introduced constrained Transformer variants that enforce identifiability in the attention layers, ensuring more unique and reliable heatmaps without sacrificing classification performance.

Parameter Non-Identifiability and Model Stealing

The problem of non-uniqueness extends beyond runtime attention maps to the fixed weights of the model itself. If different internal parameter configurations produce identical input-output mappings, it becomes impossible to uniquely reverse-engineer a model's architecture from black-box observations.

This structural ambiguity serves as a defensive shield against model-stealing attacks. Bhattamishra et al. explored this by attempting to reconstruct Transformer parameters using only queries to a black-box API. They discovered a sharp divide between single-head and multi-head architectures: while a single attention head can be uniquely recovered with a small number of queries, multi-head attention mechanisms exhibit a fundamental lack of parameter identifiability. Because different combinations of query, key, and value matrices across multiple heads can generate identical input-output behavior, proprietary multi-head configurations remain provably protected from exact reconstruction unless strict geometric constraints (such as subspace orthogonality) are imposed.

Generalizing to Feature Attribution and Saliency Maps

The identifiability problem is not unique to Transformers or attention mechanisms; it is an inherent characteristic of post-hoc explainability methods as a whole.

From a theoretical perspective, Garreau et al. analyzed feature attribution from first principles using measure theory. They proved that the mapping from the high-dimensional space of potential explanation measures down to a final, low-dimensional attribution vector is fundamentally non-bijective. Just as with attention weights, multiple distinct underlying explanations collapse into identical importance scores.

This mathematical collapse has severe practical consequences for model auditing. As Kim et al. demonstrated through controlled simulations, entirely different decision-making logics can generate virtually identical saliency maps.

Strategy A: (True Logic) --------\
                                  ===> [Identical Saliency Map]
Strategy B: (Spurious Shortcut) --/

If a model relying on a robust, causal feature and a model relying on a spurious shortcut produce indistinguishable heatmaps, a practitioner cannot rely on those heatmaps to verify model safety. A beautifully highlighted region of interest does not guarantee that the model is reasoning about that region correctly—it merely shows that the true logic and the shortcut have collapsed into the same visual footprint.

Key Foundational Work

On Identifiability in Transformers (2020) — contributed 4 concept elements to this direction.
Attention is not Explanation (2019) — contributed 3 concept elements to this direction.
Attention is not not Explanation (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attention_weight_non_identifiability_6483a892,
  title = {attention weight non-identifiability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-weight-non-identifiability/rdr_6483a892f3f34a87a1ad69bb584ef61d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_attention_weight_non_identifiability_6483a892_v1,
  title = {attention weight non-identifiability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-weight-non-identifiability/rdr_6483a892f3f34a87a1ad69bb584ef61d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake