Vision-language models turn an image into hundreds of "visual tokens," small patch embeddings that the language model must read before it answers. Recent pruning methods discard most of these tokens and keep only the ones the text prompt seems to look at. They report that almost nothing is lost. These results come from clean benchmark images. The pruning decision is itself computed from the model's attention, which is the model's internal measure of how strongly each text token relates to each image patch. That attention depends on the input distribution. When the images come from a different camera, a blind user's phone, or a resampled or blurred source, the scores used to pick tokens may spread out or move, and the pruner may throw away the evidence the model needed. If so, pruning does not just cost a fixed amount of accuracy. It amplifies the damage that distribution shift already does.
This note proposes to measure that amplification directly and to test one simple fix. Current pruners keep a fixed number of tokens. The fix is to keep however many tokens are needed to cover a fixed share of the prompt-grounded attention. On clean images that share fits in a few tokens. On shifted images, where attention disperses, the model automatically keeps more. The core claims are falsifiable. In-distribution evaluation underestimates the cost of cross-modal pruning. The extra cost is predictable without labels from how concentrated the pruning scores are. Spending tokens according to that concentration beats a fixed budget at the same average compute.
Core Claim
Define, for a pruning method at a given keep ratio, the pruning-induced shift penalty:
$$\Delta = \big(A^{\text{full}}_{\text{shift}} - A^{\text{pruned}}_{\text{shift}}\big) - \big(A^{\text{full}}_{\text{clean}} - A^{\text{pruned}}_{\text{clean}}\big),$$
where $A^{\text{full}}$ and $A^{\text{pruned}}$ are task accuracies of the unpruned and pruned model, measured on clean inputs and on their shifted counterparts. $\Delta = 0$ means pruning and shift combine additively. $\Delta > 0$ means pruning makes the model more fragile than its clean-data accuracy suggests. We hypothesize that $\Delta$ is positive and grows with the pruning ratio. We expect it to be largest for spatial shifts (resampling, blur) rather than photometric ones, and largest for prompts about small or rare objects. We also hypothesize that the mechanism is dispersion of the selection scores. Under shift, the fixed top-$k$ tokens capture a smaller share of the score mass. The proposed retention rule replaces "keep $k$ tokens" with "keep the smallest set $S$ whose normalized head-weighted scores $s_i$ satisfy $\sum_{i \in S} s_i \ge \tau$." Here $\tau$ is a target mass, calibrated on clean data so that the average number of retained tokens matches a fixed-budget baseline. This is the token-pruning analogue of nucleus sampling. Its cost is one sort and a cumulative sum over scores the pruner already computes.
Why this gap, and what prior work says
Three papers motivate the proposal by showing that cross-modal pruning works in distribution. HAP scores each attention head by how well its spatial attention tracks the prompt. It then prunes with those head-weighted scores under a fixed layer-wise budget, and retains 99.1% of LLaVA-1.5-7B performance with 32 of 576 tokens. CDPruner selects a diverse, instruction-relevant subset of tokens with a determinantal point process, which favors sets of tokens that are both relevant and dissimilar. It does not read internal attention at all. Synchrony-aware sparse attention protects the top 5% of cross-modal salient tokens in audio-visual generation, and leaves that ratio as a fixed hyperparameter. All three use a static budget, and none reports how its savings hold up when the input distribution moves. We keep HAP's scoring and change only the budget rule. This isolates the question of how many tokens to keep under shift from the question of which ones to keep.
Two papers constrain the design. Hard-coded Gaussian attention replaced self-attention with fixed input-independent patterns at no BLEU cost, but replacing cross-attention the same way cost 5 to 10 BLEU. Cross-modal selection therefore has to stay input-dependent, which rules out static spatial masks as the fix. The pruning study on long-tailed chest X-ray classifiers showed that weight pruning concentrates its errors on rare and complex cases, which were 3 to 15 times overrepresented among pruned/unpruned disagreements. That is a warning that average retained accuracy hides where the losses fall. We adopt its disagreement-set methodology for tokens.
VLM-RobustBench supplies the evidence that spatial fragility is real. Mild glass blur costs on average 8.1 points and brightness only 1.6 points, and high-severity upsampling costs up to 34 points on MMBench. It also supplies paired clean/corrupted images, which make $\Delta$ and selection stability directly measurable. HAP lists VLM-RobustBench among its 18 benchmarks, but no reported HAP result on it was found on Lacuna. So whether HAP's 99.1% retention survives corruption is, as far as this file shows, unknown.
Research Program
The work has three parts. The first is a measurement study. We take LLaVA-1.5-7B, the model on which both HAP and CDPruner report numbers, and run the full model, HAP, CDPruner, and naive pruners across keep ratios from 50% down to about 5%. Each run covers matched clean and shifted conditions, and we compute $\Delta$ per corruption family and severity. Alongside accuracy we measure selection stability: the overlap between the token sets chosen for a clean image and its corrupted twin. For geometric corruptions, token positions are mapped back to the original grid before computing overlap. We also build token-level "pruning-identified exemplars," the questions where pruned and full model disagree. We then test whether these concentrate on small-object, OCR, and counting questions, following the chest X-ray methodology.
The second part is mechanism. For every input we record the mass captured by HAP's top-$k$ tokens. We test whether this label-free quantity predicts per-example accuracy loss from pruning, and whether it drops systematically under shift. If captured mass does not move with shift, the proposed fix has no basis and we report that.
The third part is the shift-aware budget. Mass-targeted retention calibrates $\tau$ on clean in-distribution data to match HAP's average token count, and is then deployed unchanged. We run it inside the same FlashAttention-compatible scoring path HAP uses, and report wall-clock latency and KV-cache memory as well as token counts. A variable budget changes batch shapes, so we measure its real overhead rather than assume it is zero. Following the deformable attention profiling result that irregular memory access can dominate latency despite low FLOPs, we report latency at batch size 1 and at serving batch sizes.
Baselines
The naive reference points are the unpruned model, which sets the accuracy ceiling and the latency floor, and random token dropping at each keep ratio. Random dropping is instruction-blind but unbiased under shift. If it shows a smaller $\Delta$ than attention-guided pruning, that alone confirms that guided selection is the fragile part. Uniform spatial pooling to the same token count serves the same role for structured downsampling.
The strongest existing method is HAP. It reports 99.1% of original performance at 5.6% of visual tokens on LLaVA-1.5-7B, with a 2.82× speedup and 80% KV-cache reduction. We run it from its released configuration with its geometric layer budget, on identical images, prompts, and decoding settings. CDPruner is the second strong competitor. It reports 94% accuracy retention at a 95% FLOP reduction on LLaVA-1.5-7B. It is also the most important contrast, because it never reads attention and so may be inherently less shift-sensitive. Neither method receives per-benchmark tuning. Our $\tau$ is set once on clean GQA validation data and frozen, and HAP's budget is likewise fixed from its paper. Beating HAP means a smaller $\Delta$ and higher shifted accuracy at equal mean retained tokens and equal measured latency on the pooled clean-plus-shifted test set, with clean accuracy within noise of HAP's.
Two ablations isolate the key ingredient. The first, a variance control, gives each input a random budget drawn from the same distribution of budgets that mass-targeting produces. It tests whether gains come from allocating tokens to the right inputs or merely from varying the budget. The second, an oracle, gives each input the smallest budget at which the pruned answer matches the full model's. It bounds how much any budget rule could recover. A third run applies mass-targeting to uniformly head-averaged scores. It tests whether the rule depends on HAP's head weighting, which HAP argues is necessary because averaged heads are dominated by background sinks.
Why not just…?
Why not just use a larger fixed budget? Keeping 15% of tokens instead of 5% will certainly shrink $\Delta$. The question is whether it does so more cheaply than adaptive allocation. Everything here is compared at matched mean tokens and matched latency, as full accuracy-versus-cost curves. If a fixed budget lies on or above the adaptive curve, the adaptive rule is unnecessary and we will say so. That outcome is plausible if dispersion under shift turns out to be uniform across inputs.
Why not just use CDPruner, which never reads attention? This is a real threat. If CDPruner's diversity term already keeps peripheral tokens that matter under shift, its $\Delta$ may be near zero, and the problem we target would be specific to attention-score pruners. The measurement study settles this first. If CDPruner is shift-robust, the contribution narrows to a diagnosis of attention-guided pruning plus a fix for that family. We would also test mass-targeting on CDPruner's relevance scores, to see whether it transfers.
Why not just make the base model robust, by fine-tuning on corrupted data? Robust fine-tuning lowers the full model's shift drop, but it does not by itself make the pruner's selection stable. It can also be combined with any pruning rule. We run HAP and our rule on top of the base model after a short corruption-augmented fine-tune on held-out corruption types. If $\Delta$ vanishes after fine-tuning, the budget rule is redundant. The training-free setting the pruning literature targets would still make a no-retraining fix useful.
Why not just protect a fixed top fraction of salient tokens, as in the audio-visual work? Protection keeps a fixed fraction regardless of how dispersed the scores are. Under our hypothesis, that is exactly the failure mode, because under shift the important tokens may lie outside any fixed top fraction. We include a 5%-protected variant of HAP as an additional baseline.
Proposed Experiments
Synthetic shift comes from VLM-RobustBench's paired corruptions over MMBench and its other source sets, reported per corruption family and severity. Natural acquisition shift comes from benchmarks in HAP's suite whose images differ in capture conditions. We treat VizWiz, taken by blind users, and RealWorldQA as shifted targets relative to GQA and TextVQA as the calibration distribution. Grounding on RefCOCO probes whether small referred objects are lost first. The model is LLaVA-1.5-7B, with LLaVA-NeXT as a replication.
The decisive checks are three, in order. The first asks whether $\Delta$ is significantly positive for HAP at aggressive keep ratios, using paired bootstrap over questions. If it is not, the main result is that cross-modal pruning is shift-safe, which is still worth reporting. The second asks whether captured top-$k$ mass falls under shift and correlates with per-example pruning loss. The third asks whether mass-targeted retention beats both a fixed budget and the random-budget control at matched mean tokens and latency on the pooled test set, without losing clean accuracy. Secondary analyses report selection-stability overlap and the composition of pruning-identified exemplars. These indicate whether gains come from rescuing small-object and text-reading questions, the cases the chest X-ray study suggests pruning harms first.
References
- HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
- Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention
- Hard-Coded Gaussian Attention for Neural Machine Translation
- How Does Pruning Impact Long-Tailed Multi-label Medical Image Classifiers?
- VLM-RobustBench: A Robustness Benchmark for Vision-Language Models
- DEFA: Efficient Deformable Attention Acceleration via Pruning-Assisted Grid-Sampling and Multi-Scale Parallel Processing