Identifiability and the Null Space of Transformer Attention

This direction is split into other directions: attention weight non-identifiability. Its last version stays readable and citable here.

Version 1 of 112 papers28 conceptsfrozen 2026-10-04

We often treat Transformer attention maps as "windows" into a model's reasoning. When a sentiment classifier highlights the word "excellent," we assume that specific weight is the reason for the positive prediction. However, a growing body of research suggests this intuition is mathematically fragile. Because of a property called non-identifiability, there are often infinitely many different attention maps that produce the exact same model output.

The Geometry of the Null Space

The core of the problem lies in the dimensionality of the attention mechanism. In a standard Transformer head, attention weights are used to compute a weighted sum of "Value" vectors. If you have a sequence of length $n$ and a head dimension $d$, you are essentially using $n$ weights to create a vector in a $d$-dimensional space.

As Gino Brunner, Roger Wattenhofer, and colleagues point out in On Identifiability in Transformers, a problem arises whenever the sequence length $n$ is greater than the dimension $d$. In this scenario, the mapping from attention weights to the output is non-injective.

Mathematically, this creates a null space. You can change the attention weights—shifting focus from one word to another—and as long as those changes stay within the null space, the final representation remains identical. This means the "heatmap" we see is just one arbitrary point in a large space of possible explanations.

Attention is Not Necessarily Explanation

If multiple attention distributions yield the same result, can we trust any single one of them? Sarthak Jain and Byron C Wallace famously argued in Attention is not Explanation that attention weights are often uncorrelated with other measures of feature importance.

They demonstrated that one can find "adversarial" attention distributions: configurations that look completely different to a human observer but result in the same prediction. This lack of uniqueness suggests that the visual focus of a model does not necessarily represent the underlying causal factors of its decision. Adrien Bibal and co-authors further synthesized this in their survey of the debate, noting that if a model can reach the same conclusion while highlighting different features, a single heatmap cannot be a definitive explanation.

The Challenge of Multi-Head Ambiguity

The problem compounds in multi-head architectures. While a single-head attention layer might be easier to analyze, Satwik Bhattamishra and Varun Kanade show in Provably Learning Attention with Queries that multi-head attention is fundamentally harder to "unmask."

Because different heads can compensate for one another, there is no unique solution for the individual head parameters that can be recovered from black-box observations. This structural ambiguity means that even if we could perfectly observe the input-output behavior of a Transformer, we still couldn't uniquely determine what each head is "doing" without additional constraints like subspace orthogonality.

Why This Matters for Discovery

This isn't just a theoretical concern for interpretability; it affects how we use Transformers for scientific discovery. For instance, Lu Mi and J. Nathan Kutz explored using attention to map causal relationships in biological neurons in Causalformer. They found that the lack of identifiability can lead to inconsistent results across different training runs. If the model can capture multiple valid predictive patterns to explain the same data, the resulting "wiring diagram" might reflect a random internal configuration rather than the ground-truth biological topology.

Similarly, Kaiyue Wen and Andrej Risteski argue in Transformers are uninterpretable with myopic methods that looking at individual components in isolation is misleading. Models can solve complex tasks using "messy," non-interpretable internal patterns just as effectively as human-readable ones.

Moving Toward Identifiable Models

To address these issues, researchers like Rishabh Bhardwaj and Soujanya Poria have begun exploring ways to build More Identifiable yet Equally Performant Transformers. The goal is to constrain the model's weight space so that the resulting attention maps are unique, ensuring that the "explanation" we see is the only one the model could have used to reach its conclusion. Until such constraints become standard, attention maps should be viewed as possible justifications rather than unique causes.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_identifiability_and_the_null_space_of_tr_1493dffd,
  title = {Identifiability and the Null Space of Transformer Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identifiability-and-the-null-space-of-transformer-attention/rdr_1493dffdb89b45c6b43fb6c31e009bd6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_identifiability_and_the_null_space_of_tr_1493dffd_v1,
  title = {Identifiability and the Null Space of Transformer Attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identifiability-and-the-null-space-of-transformer-attention/rdr_1493dffdb89b45c6b43fb6c31e009bd6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}