Overcoming Positional Bias in Multimodal Token Pruning

23 concepts16 papers1 proposal

As Multimodal Large Language Models (MLLMs) scale to handle high-resolution images and long video sequences, they face a severe computational bottleneck. Processing thousands of visual tokens dramatically inflates the Time-to-First-Token (TTFT) and overwhelms the memory capacity of the KV cache, as noted by Hengrui Hu et al. in EAKV: An Entropy-Driven Adaptive KV Compression Framework for Long Video Understanding.

To accelerate inference, a natural strategy is token pruning: identifying and discarding "unimportant" visual tokens before they pass through the heavy language model layers. Intuitively, the internal attention weights of the transformer seem like the perfect metric for this task. If a visual token receives high attention scores from text queries or surrounding context, it must be semantically critical.

However, recent evaluations have revealed a startling flaw in this assumption. Zichen Wen et al. demonstrated in Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem? that sophisticated attention-based pruning methods frequently perform worse than simply dropping tokens at random.

The reason raw attention scores fail as a proxy for importance is that they do not purely reflect semantic content. Instead, they are heavily distorted by positional bias—an artifact inherited from text pre-training and exacerbated by 1D positional encodings.


The Text Pre-Training Artifact: Attention Sinks

Why do attention scores lie? The root of the issue lies in the language model backbone. During text-only pre-training, transformers learn to use specific sequence positions—most notably the very beginning and the very end of a sequence—as structural anchors or "attention sinks."

As explained by Xin Zou et al. in Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention, these boundary positions receive disproportionately high attention scores regardless of their actual semantic content because they serve a syntactic role in text processing.

Text Pre-training Bias:
[Start Token] ---> (Absorbs excess attention / acts as structural anchor)
[End Token]   ---> (Receives high baseline attention due to sequence boundaries)

When a 2D image is converted into visual tokens and injected into a language model, it inherits this text-based structural bias. If the visual inputs are not perfectly aligned with these low-index structural positions, the model fails to prioritize visual features correctly, as shown by Jie Huang et al. in Revisiting Multimodal Positional Encoding in Vision–Language Models.

Furthermore, Jingyu Liu et al. point out in Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation that these sink token and proximity biases create artificial "hotspots" in the attention matrix, blinding naive selection algorithms to the true distribution of information.


The Geometry of Flattening: Why Pruning Hates the Top of Images

The conflict between 2D visual space and 1D sequence processing becomes even more destructive when interacting with modern positional encodings like Rotary Position Embeddings (RoPE).

To feed a 2D image into a language model, it must be flattened into a 1D sequence. This is typically done via a raster scan (row by row, from top-left to bottom-right). Consequently, tokens representing the top of the image appear early in the sequence, while tokens representing the bottom appear at the end.

2D Image Grid:                 1D Flattened Sequence:
[ Row 1 (Top)    ]  Raster     [ Row 1 ] [ Row 2 ] [ Row 3 ] [ Row 4 ]
[ Row 2          ]  Scan       (Early)                       (Late)
[ Row 3          ]  ---------> 
[ Row 4 (Bottom) ]             RoPE Long-Term Decay / End-of-Sequence Bias
                               [Artificially Low] <------- [Artificially High]

In Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration, Mark Endo et al. uncovered a critical side effect of this mapping: attention mechanisms utilizing RoPE exhibit a long-term decay property that inherently favors tokens appearing later in the sequence. Because the bottom of the image maps to the end of the sequence, the model over-attends to the bottom rows and assigns artificially low attention scores to the top rows.

When an attention-based pruning algorithm runs, it blindly follows these skewed scores. The result is a severe top-heavy pruning bias, where the top half of an image is systematically erased regardless of what it contains. This loss of holistic context severely degrades performance on vision-centric tasks that require precise spatial awareness, such as object tracking and localization, a vulnerability highlighted by Yinchao Ma et al. in Learning Generalized Trackers with Elastic Token Budgets and Evelyn Zhang et al. in D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning.


Solutions: Debiasing, Decomposition, and Redundancy

To build pruning frameworks that actually preserve performance, researchers are developing training-free methods to bypass or correct these positional artifacts.

1. Explicit Positional Correction

One straightforward solution is to calculate the baseline positional bias of the decoder and subtract it from the attention scores before pruning. In PosPrune: Visual Token Pruning with Positional Bias Correction for Efficient Large Vision-Language Models, Ziyang Wang et al. demonstrate that correcting for this inherent decoder bias allows the model to balance token relevance with spatial diversity.

Similarly, kaiyuan Li et al. introduce a rebalancing mechanism in Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization to ensure that informative tokens at the beginning of a sequence are not prematurely discarded.

2. Bypassing Attention via Matrix Decomposition

Instead of trying to fix flawed attention scores, some approaches isolate semantic relevance using algebraic alternatives. In Vision Token Pruning via Query--Vision Interaction Decomposition, Harshithanjani Athi et al. apply Singular Value Decomposition (SVD) directly to the interactions between text queries and image features. By decomposing this matrix, they extract the core semantic relationships without triggering the positional biases embedded in standard attention layers.

3. Shifting from Importance to Redundancy

A final paradigm shift abandons the search for "important" tokens entirely, focusing instead on eliminating redundancy. Lei Du notes in AdaV: Adaptive Text-visual Redirection for Vision-Language Models that filtering tokens through internal attention weights is fundamentally bottlenecked by sequence-position biases.

To circumvent this, Zichen Wen et al. introduce DART in Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More. Rather than relying on attention saliency, DART identifies and removes duplicated visual information. By focusing on similarity and redundancy rather than absolute importance, the method avoids positional traps entirely, achieving significant speedups while preserving the model's underlying accuracy.

Key Foundational Work

Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration (2025) — contributed 5 concept elements to this direction.
Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention (2025) — contributed 3 concept elements to this direction.
D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning (2026) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attention_based_pruning_bias_e67339c3,
  title = {attention-based pruning bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-based-pruning-bias/rdr_e67339c336ff4310a81d62f5ef8d2524},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_attention_based_pruning_bias_e67339c3_v1,
  title = {attention-based pruning bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-based-pruning-bias/rdr_e67339c336ff4310a81d62f5ef8d2524/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake