In an ideal Transformer, the attention mechanism acts like a spotlight, illuminating the most relevant information regardless of where it sits in a sequence. However, a growing body of research reveals that this spotlight is often "lazy." Instead of seeking out semantic meaning, Transformers frequently focus on tokens based on their physical position—favoring the very beginning, the very end, or tokens immediately adjacent to a query.
This positional bias creates a fundamental disconnect: a token’s attention score often reflects its structural location rather than its semantic importance. Understanding and correcting this bias is now central to making Large Language Models (LLMs) and Vision-Language Models (VLMs) more efficient, truthful, and capable of handling long contexts.
The Structural Trap of Causal Masking
The most pervasive bias stems from the architecture itself. In autoregressive models, "causal masking" ensures that a token can only look at what came before it. As Xiangming Gu and Prannay Kaul observe, this creates a structural "attention sink" at the very first token.
Because the first token is visible to every subsequent query, it becomes a convenient "trash bin" for residual attention scores when the model finds nothing else relevant to attend to. This isn't because the first token is meaningful; it’s because it’s the only position that is always there. Zewen Qiang shows that this "initial saliency" compounds across layers, leading to a primacy bias where the model over-relies on the start of a prompt and ignores the middle—the "lost-in-the-middle" phenomenon.
Mathematical Artifacts and Spatial Blindness
In Vision-Language Models, the problem becomes spatial. When a 2D image is flattened into a 1D sequence of tokens, the model’s positional encoding scheme—often Rotary Positional Embeddings (RoPE)—introduces a mathematical decay. Mark Endo and Qizhe Zhang have demonstrated that this causes the model to disproportionately attend to tokens at the end of the sequence.
In practice, this means the model might obsess over the bottom of an image simply because those pixels appear last in the sequence, while completely ignoring the top. Bozhi Luan notes that this architectural bias is so strong that the model's "attention maps" often follow fixed geometric patterns regardless of what is actually in the picture.
The Failure of Attention-Based Pruning
This bias isn't just a theoretical nuisance; it breaks the tools we use to make models faster. To speed up inference, researchers often use "token pruning"—throwing away "unimportant" tokens to save memory. If we use raw attention scores to decide what to keep, the bias leads to disaster.
As Zichen Wen and kaiyuan Li point out, attention-based pruning often accidentally discards critical information from the middle of a text or the top of an image because those tokens have lower "structural" scores. Zhuoming Chen further argues that deterministic "Top-K" selection is a biased estimator that fails on tasks requiring global information aggregation, like summarization, because it ignores the "long tail" of low-score tokens that collectively carry significant weight.
Decoupling Content from Structure
To fix this, researchers are developing "calibration" techniques that isolate the true semantic signal from the positional noise.
1. Zeroing Out the Bias
One intuitive approach is to measure the model's "background noise." Heng Guo and the authors of Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration suggest feeding the model a "semantic-free" input—like a black video frame or a blank string. Any attention patterns that emerge on a blank input are purely structural. By subtracting this "bias map" from the attention scores of a real input, the model can finally "see" the actual content.
2. Normalizing for Opportunity
Since early tokens accumulate more attention simply by being in the sequence longer, Yefei He and Zhonghua Jiang propose normalizing scores by the number of "attention opportunities" a token has had. This levels the playing field, allowing informative tokens that appear later in a long document to compete fairly with the "attention sinks" at the beginning.
3. Shifting to Intrinsic Metrics
Perhaps the most robust solution is to stop relying on attention scores entirely for importance estimation. Zichen Wen introduces V²Drop, which looks at "intrinsic variation"—how much a token's representation actually changes as it passes through layers. If a token doesn't change, it's "lazy" and can be pruned, regardless of its position. Similarly, Qizhe Zhang suggests using visual-only cues from the encoder rather than cross-modal attention to identify essential image patches.
Towards Robust Understanding
The goal of this research direction is to move from "position-aware" models to "content-aware" ones. By calibrating for U-shaped biases Cheng-Yu Hsieh and neutralizing the recency effects of RoPE Jiaming Li, we can build models that don't just look at the most "convenient" tokens, but the most meaningful ones. As Liang CHEN demonstrates, making representations "order-insensitive" is the key to ensuring that an AI's personality and knowledge remain stable, no matter how you phrase the prompt.