Multimodal language models turn every image into hundreds of "visual tokens," the patch embeddings the language model reads alongside the text. Feeding many images, such as a photo album or a set of retrieved frames, makes these models slow. It also makes them worse: accuracy falls steadily as irrelevant distractor images are added. The usual fix for speed is to prune tokens, keeping those the model's own attention marks as important. Attention, though, is a poor measure of relevance. Much of it depends on where a token sits in the sequence and on salience that does not depend on the question at all. Pruning therefore tends to keep the wrong tokens, and random dropping often does as well.
This note proposes a simple way to score each visual token. We measure how much more the actual question attends to it than a content-free "null" question does, on the same input. Subtracting the null question's attention cancels positional artifacts and generic salience in one step. The same subtraction removes the component shared by all text queries, which is the query side of the vision–text "modality gap." What is left is question-specific relevance. The spread between two different null questions gives a noise floor, so the model can keep only tokens clearly above it. The token budget then shrinks when the context is mostly distractors and grows when many tokens matter. We test whether this makes pruning a way to remove distractor noise in multi-image settings, and not only a speed trick.
Core Claim
Several findings about attention-based pruning point to the same problem. Feather the Throttle shows that the long-range decay in rotary position embeddings makes FastV-style pruning discard the top rows of images. D²Pruner shows that the shape of this bias depends on the architecture: bottom-heavy in LLaVA, focused on the periphery in InternVL and Qwen-VL. Token Pruning in MLLMs: Are We Solving the Right Problem? finds that random dropping and pooling beat attention pruning on GQA and POPE. Current fixes subtract a fixed correction estimated offline. PosPrune fits a degree-3 polynomial on 800 MME samples (PosPrune), and D²Pruner divides by a bias prior averaged over COCO. Both were tested only on single images.
We claim that the bias worth removing depends on the input. In a sequence of many images, position effects build up across images, not only within one. Each image also has its own generic salience, and the query shares a large text-wide component with every other query. A contrast computed per input, between the real question and a null question over the same visual tokens, removes all three. Let $\bar q$ be the mean pre-rotary query vector of the question tokens at a shallow layer $\ell$. Let $\bar q_0$ be the same quantity for a generic prompt such as "Describe the images." Let $k_i$ be the pre-rotary key of visual token $i$, with head dimension $d$. The score is
$$ s_i = \frac{(\bar q - \bar q_0)^\top k_i}{\sqrt{d}}. $$
A second generic prompt gives $\bar q_0'$. Its contrast scores $n_i = (\bar q_0' - \bar q_0)^\top k_i/\sqrt{d}$ estimate how much scores move with no change in meaning. We keep token $i$ when $s_i > z\,\hat\sigma_n$, where $\hat\sigma_n$ is the standard deviation of $n_i$ over the input's visual tokens and $z$ is a single global constant. Images are then allocated budget in proportion to their above-threshold score mass.
This connects to the modality gap in a specific way. A constant visual-mean offset in the keys cannot change how tokens rank for a fixed query. The text-side shared direction can, because it interacts differently with each key. Subtracting $\bar q_0$ removes exactly that direction.
The idea borrows from text-only work on distraction. Distraction-Aware Answer Selection (DAS) rescores answers by how likely each document is to generate the query. That is a relevance-over-baseline contrast, which we move from whole documents to visual tokens. Superposition Prompting evaluates document branches in parallel against a shared prefix. We reuse this fork structure so that the null prompts read the already-computed visual key cache. The overhead is then a few extra query vectors at one layer, and no full attention map is needed. This matters because the need for full attention maps is what breaks FlashAttention, the fused attention kernel, for FastV and slows SparseVLM (Token Pruning in MLLMs).
Research Program
The first step is diagnostic. It tests whether the problem exists in the multi-image regime, which no work in the file has checked. We use LOCOVQA-style interleaved sequences (Losing Visual Needles in Image Haystacks) and Visual Haystacks (Visual Haystacks), with the image that answers the question (the needle) placed at a controlled index among $N$ images. We then measure needle-token recall, the fraction of the needle's tokens that survive pruning, for FastV, FEATHER's rotary-free scoring, and D²Pruner-style prior subtraction.
We expect bias to build up across images. On LLaVA-OneVision, needles placed early should lose far more tokens than needles placed late. The direction should differ by backbone, following the architecture-dependent patterns reported by D²Pruner and the start and end preferences of GPT-4V and Mantis in LOCOVQA.
In the same analysis we split the variance of raw pre-rotary scores across visual tokens into two parts. One comes from the component shared across many generic and real questions. The other comes from the question-specific residual. This tells us how much of "attention importance" is actually question-independent. The analysis is the empirical core of the modality-gap claim. It also explains why QViD, which runs a low-rank decomposition of raw $Q_{txt}K_{vis}^\top$, might capture shared, question-independent structure instead of task semantics.
The second step is the method: null-query contrast, the noise-floor threshold, and image-level budget allocation. It stays training-free and uses the same shallow layer across models, chosen once on a held-out split. Inside each image, the surviving tokens can be passed through duplicate removal in the style of DART (Stop Looking for "Important Tokens"). This combines the two sides of the tension in the file. The contrast removes diverse but irrelevant distractors, which DART cannot see because it ignores the question. DART then removes redundancy within the relevant images.
Baselines
The do-nothing reference is the unpruned model on the same inputs. It gives the accuracy ceiling for noise-free processing and, on distractor-heavy inputs, a possible floor that pruning can exceed. Random dropping and 2×2 pooling at matched token counts are the naive references. The audit in Token Pruning in MLLMs shows they beat attention pruning on GQA and POPE, so any method that loses to them fails.
The strongest multi-image competitor is AVTP (Multi-Image Visual Token Pruning). It scores tokens by how much their hidden states change across layers and gives more budget to more relevant images. It reports that Qwen3VL-8B keeps 96.1% and InternVL3.5-8B keeps 94.1% of original accuracy with 50% of visual tokens and a 2× speedup on MuirBench, MIRB, BLINK, Qbench2 and NLVR2. It is the right opponent because it already does image-level budget allocation and is FlashAttention-compatible. We run it with its published layer and scaling settings, plus a tuning sweep of the same size as ours on the same held-out split. We beat it if we retain more accuracy at 50% tokens and at 20% tokens, at equal measured wall-clock prefill time, on the same backbones.
For debiasing, the strongest competitor is D²Pruner, which reports 85.7% localization performance at 90% token reduction on InternVL-2.5-8B. It is the closest static version of our idea. Its dataset-wide prior removes average positional bias but cannot adapt to the particular images in a given input. We rerun it with its COCO prior and with a prior re-estimated on multi-image inputs, so that it is not handicapped by being calibrated on single images.
On Visual Haystacks, the image-level retrieval pipeline MIRAGE reports up to a 13% improvement over open-source multimodal models. We use its query-aware retriever as a baseline that filters whole images and feeds all tokens of the kept images. For token pruning methods on Visual Haystacks, no reported result was found on Lacuna. The audit paper evaluated them there but the file quotes no numbers, so we will produce those numbers ourselves.
Two ablations isolate the key ingredient. The first uses raw pre-rotary scores $\bar q^\top k_i$ with the same threshold and allocation. This is essentially FEATHER's criterion, and it tests whether the null subtraction matters. The second replaces the per-input null with a fixed dataset-averaged null query, which tests whether computing the null on each input matters. A third ablation swaps the adaptive threshold for a fixed budget at matched average token count, which tests whether adaptivity itself helps.
Why not just…?
Why not just subtract a fixed bias prior, as D²Pruner or PosPrune do? This is the most serious threat. If positional bias in multi-image sequences is mainly a fixed function of absolute index, a prior re-estimated on multi-image data would match our per-input contrast. The dataset-averaged-null ablation decides this. If it comes within noise of the per-input version on Visual Haystacks and LOCOVQA, we will report that the gain comes from calibrating on multi-image data, not from per-input contrast. We would then drop the per-input claim and keep only the adaptive threshold.
Why not just retrieve the right image with CLIP and drop the rest? For single-needle retrieval this may be enough, and MIRAGE's gains on Visual Haystacks show that image-level filtering is strong. It fails where answers need several images at once, such as comparison tasks in NLVR2 and MuirBench. It also leaves background clutter inside the kept image, which CROP (CROP) shows can be pruned heavily while keeping 96% of performance. We report results separately for single-needle and multi-image-reasoning tasks so that the regime where retrieval suffices is visible.
Why not just tune LayerNorm to close the modality gap and then use raw attention? Tuning LayerNorm in Attention shows that tuning as little as 0.004% of parameters realigns visual representations and cuts POPE hallucination. It never tests whether this removes positional bias in attention. We run the diagnostic from the research program on a LayerNorm-tuned LLaVA. If needle recall under raw attention scoring becomes flat across positions, then a small amount of training fixes the problem more cleanly than inference-time contrast, and we would say so.
Why not just prune randomly or pool? Random dropping and pooling spread tokens evenly by construction, and on coarse benchmarks this is enough. With 36 images and a single small needle, keeping 10% of tokens uniformly keeps 10% of the needle. Our method should keep most of it. The needle-recall curves make this comparison directly measurable.
Proposed Experiments
The backbones are LLaVA-OneVision-7B, InternVL (via opengvlab/internvl) and a Qwen-VL model. These cover the three bias shapes reported by D²Pruner.
The primary benchmarks are Visual Haystacks, and LOCOVQA in both composed-grid and interleaved form with 1 to 36 images. LOCOVQA reports drops of more than 50% on OK-VQA over that range. Secondary benchmarks are MuirBench, MIRB, BLINK and NLVR2 for multi-image reasoning. RefCOCO and POPE serve as single-image sanity checks, so that the method does not give up the localization gains of FEATHER and D²Pruner.
The metrics are task accuracy, needle-token recall, measured prefill wall-clock time, and kept-token count as a function of distractor count.
Three results would decide the question. First, needle-token recall under our method should be flat across needle position, while FastV and raw pre-rotary scoring should show a monotone positional trend. If the trend is absent in raw scores, the problem our method solves does not exist in this regime. Second, on LOCOVQA the slope of accuracy against the logarithm of the number of distractors should be less steep than for the unpruned model. That is, pruning should reduce distraction, not only cost. If the pruned model merely tracks the unpruned curve, the noise-sensitivity claim fails. Third, the adaptive threshold should keep far fewer tokens as distractors grow while accuracy stays flat, and it should beat a fixed budget at matched average token count. If it does not, the noise-floor idea adds nothing beyond per-image allocation.
References
- Feather the Throttle: Revisiting Visual Token Pruning for Vision-Language Model Acceleration
- D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
- Token Pruning in Multimodal Large Language Models: Are We Solving the Right Problem?
- PosPrune: Visual Token Pruning with Positional Bias Correction for Efficient Large Vision-Language Models
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
- Vision Token Pruning via Query--Vision Interaction Decomposition
- Stop Looking for "Important Tokens" in Multimodal Language Models: Duplication Matters More
- Multi-Image Visual Token Pruning in Large Visual Language Models
- CROP: Contextual Region-Oriented Visual Token Pruning
- Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning