Vision-language models turn an image into hundreds or thousands of "visual tokens", one per image patch. Visual tokens often make up 97–99% of the model's input, as CARES: Context-Aware Resolution Selector for VLMs reports. A popular way to cut this cost is to score each patch by how relevant it looks to the text question and keep only the top ones. The saved budget can then be spent on something useful, most obviously on feeding the model a higher-resolution image before pruning. Many of these relevance scores are cosine similarities between image-patch embeddings and a text embedding from a CLIP-style model, meaning a model trained to place matching images and captions near each other. Such models have a well-documented "modality gap": image embeddings and text embeddings sit in two separate regions of the shared space. Their cross-modal similarities therefore bunch into a narrow band, and part of each score reflects which region a vector lives in rather than what it depicts.
This proposal claims that the modality gap quietly degrades text-guided token pruning, and that fixing it matters most exactly when saved compute is reinvested in resolution. The fix is to remove each modality's characteristic offset, estimated separately for each input resolution, before scoring. At higher resolution there are many more candidate patches competing for the same small set of kept slots, so a noisy relevance score does more damage. Our prediction is that "high resolution, then prune with calibrated scores" beats both "low resolution, dense" and "high resolution, then prune with raw scores" at an equal number of tokens entering the language model. We further predict that the gain comes from a rank-changing correction, not from simply rescaling scores.
Core Claim
Let $v_i$ be the embedding of visual token $i$, and let $t$ be the text embedding of the instruction, both unit-normalized in the contrastive space. Most text-guided pruners rank tokens by $v_i^\top t$ or its cosine. I0T: Embedding Standardization Method Towards Zero Modality Gap shows that each modality carries a fixed mean offset driven by a few spiking coordinates. Write $\mu_t$ for the mean text embedding. Then
$$v_i^\top t = v_i^\top (t-\mu_t) + v_i^\top \mu_t .$$
The second term does not depend on the question. It measures how much patch $i$ happens to align with the generic "text-ness" direction, and it varies from patch to patch. Because of this, it reorders the ranking, not just its scale. We propose scoring with
$$q_i^{(r)} = \cos\big(v_i-\mu_v^{(r)},\; t-\mu_t\big),$$
where $\mu_v^{(r)}$ is the mean visual token measured at resolution regime $r$, for example a full 336-pixel view or one tile of a high-resolution multi-tile input. Centering the text side removes the question-independent bias. Centering the visual side with statistics matched to the resolution handles a separate problem: the visual token distribution shifts as resolution and tiling change. The file flags this shift as a reason fixed standardization may break on dynamic inputs.
The claim is testable in three parts. First, calibrated scores place more kept tokens inside the image region the text actually refers to. Second, this advantage grows with input resolution when the kept budget is held fixed. Third, simply rescaling the raw scores with a temperature or exponent does not reproduce the gain. The third part is what separates a geometric explanation from a tuning knob.
Relation to Prior Work
Three pruning methods motivate the work, and their shared assumption is the target. TRIPS uses the text encoder's summary token to select image patches inside the vision backbone and reinvests the savings into 456-pixel inputs. CDPruner weights a determinantal point process by text-to-patch cosine relevance. A determinantal point process is a sampler that prefers subsets of tokens that are both relevant and mutually different. VL-Cache ranks visual cache entries by how instruction tokens attend back to them. All three take cross-modal similarity at face value. None measures the gap or corrects for it.
Three gap papers supply the method, and one of them also constrains it. Mind the Gap explains why the gap exists: each encoder funnels its outputs into its own narrow cone, and the contrastive loss holds the two cones apart. I0T shows that subtracting modality means collapses the centroid distance from 0.7642 to 0.0102. Closing the Modality Gap Aligns Group-Wise Semantics shows that global instance retrieval is largely insensitive to the gap because it depends only on rank.
That last result is the main constraint. It predicts that centering should do little whenever a pruner ranks tokens against a fixed text vector using a correction that preserves rank. Our derivation above says the text-side correction is not rank-preserving at the patch level. The experiments test that directly rather than assume it. We avoid gap-closing methods that retrain the encoders, such as ModGap and AlignCLIP. Our point is a training-free fix that can be added to existing pruners. AlignCLIP's warning that naive shifting distorts structure within a modality matters less here, because we only use the centered vectors to score tokens. The tokens passed to the language model are left unchanged.
On reinvestment, CARES picks one resolution per query, and RECLIP uses low resolution as a training schedule. We instead hold the number of tokens entering the language model fixed and ask which input resolution, combined with which selector, spends that budget best.
Research Program
The work has three stages: diagnose, correct, and reinvest.
The first stage is diagnostic. On LLaVA-1.5-7B and LLaVA-NeXT, we compute patch-to-instruction similarities in the CLIP space that CDPruner-style relevance uses. We record their spread, and we measure the share of variance explained by the question-independent term $v_i^\top\mu_t$. We also measure how much $\mu_v^{(r)}$ drifts across native resolution and individual tiles. For grounding we use Flickr30k Entities, which pairs phrases with bounding boxes, and RefCOCO. On these we define a selection hit rate: the fraction of kept tokens that fall inside the referred box, at a fixed keep count $k$.
The second stage applies the correction above to three scoring rules: top-$k$ relevance, the relevance term inside CDPruner's diversity-weighted sampler, and TRIPS-style text-guided fusion run inference-only. The mean vectors come from a few thousand unlabeled images and captions for each regime $r$, so calibration costs essentially nothing.
The third stage traces cost–accuracy curves. We fix the number of tokens entering the language model at 32, 64, and 144. We then vary the input from native 336-pixel resolution to multi-tile high resolution and prune down to the fixed count with each selector.
Baselines
The naive reference point is the dense model at native resolution, together with dense low-resolution inputs that are downsampled until they yield the same token count. These bracket what pruning has to beat. Random token dropping at the same count, in the spirit of PatchDropout, shows whether any selector matters at all.
The strongest existing competitors are HAP and CDPruner. HAP picks visual tokens using the language model's own attention heads, weighted by how well each head grounds the prompt. It retains 99.1% of original performance on LLaVA-1.5-7B while keeping 32 of 576 tokens. CDPruner reports 94% accuracy retention at a 95% FLOP reduction on LLaVA-1.5-7B, and 100.1% at 77.8% pruning on LLaVA-NeXT. HAP is the hardest to beat at tiny budgets, and CDPruner is the direct host for our correction. We run both from their released settings on the same checkpoints and benchmarks, at matched token counts and matched latency measured on the same GPU. Any temperature or keep-schedule hyperparameters get the same tuning budget we give ourselves, on a held-out split. No reported result was found on Lacuna for either method on high-resolution-then-prune at a fixed token count, so we will produce those numbers ourselves.
Beating these baselines means that, at 32 or 64 kept tokens, the high-resolution input with calibrated CDPruner exceeds HAP and raw CDPruner run on the same input. The margin must be outside seed variance on text-heavy benchmarks, and the latency must not exceed theirs.
The ablations isolate the key ingredient. We run text-side centering only, visual-side centering only, a single global visual mean versus means matched to each resolution, and a rank-preserving rescaling of the raw scores. The rescaling is described below.
Why not just…?
Why not just tune the temperature of CDPruner's relevance term? This is the most serious threat. The modality gap compresses scores into a narrow band, and a sharper exponent would re-expand that band and restore the balance between relevance and diversity without any geometry. Our derivation predicts that rescaling cannot remove the question-independent ordering term, so it should recover part of the gain but not all of it. We give the temperature a grid search as large as our own tuning budget. If tuned rescaling matches calibrated scoring on hit rate and task accuracy, we will report that the gap problem is only a scale problem and drop the geometric claim.
Why not just apply I0T's global post-hoc centering? That is the simplest form of our method, and we include it as an ablation. I0T estimates its means from batches of whole images at one resolution. The open question is whether tiles and higher resolutions shift the visual mean enough to matter. The diagnostic drift measurement and the global-versus-per-resolution ablation decide this. If global centering suffices, we adopt it and the contribution becomes the reinvestment result.
Why not just use the language model's attention, as HAP does, and avoid the contrastive space entirely? Attention-based selection has to run the visual tokens through at least part of the language model. That limits savings at high resolution, where the token count before pruning is the bottleneck. VL-Cache shows that attention across the modality boundary has its own quirks, concentrating on a few heavily attended tokens. We compare directly at matched end-to-end latency. If HAP wins at equal latency, that bounds the value of pre-language-model pruning.
Why not just pick a lower resolution per query, as CARES does? Choosing a resolution and choosing tokens address different decisions. CARES cannot keep the fine detail of a small text region while discarding the rest of the image. We include a CARES-style oracle that selects the best resolution per query from our grid, run densely. Calibrated pruning should beat it on document-like questions; it may not on whole-scene questions.
Proposed Experiments
The evaluation uses the text-heavy benchmarks on which VL-Cache reports about 98% retention: DocVQA, TextVQA, and ChartQA. It also uses MME, MMBench, and RefCOCO from HAP's suite, plus Flickr30k Entities and RefCOCO for hit rate.
Because Closing the Modality Gap shows that rank-based metrics hide damage that only distance-based metrics reveal, we add a group-level check. We pool the kept tokens of each image and measure clustering V-Measure on MSCOCO, a score of how well clusters match the true categories. This tests whether calibrated pruning preserves semantic structure that accuracy does not reveal.
Three outcomes decide the main claim. First, calibrated scoring must raise box hit rate over raw scoring at fixed $k$ and also over the best-tuned temperature. Second, the accuracy advantage of calibrated over raw selection must grow from native to tiled high-resolution inputs at a fixed count of 64 tokens. Third, the best high-resolution calibrated configuration must beat dense low-resolution input at equal token count on DocVQA and TextVQA. If the first check fails, the claimed mechanism is wrong. If the first holds but the second does not, the correction is real but has nothing to do with reinvestment, and the paper becomes a smaller one about pruning hygiene.
References
- CARES: Context-Aware Resolution Selector for VLMs
- I0T: Embedding Standardization Method Towards Zero Modality Gap
- TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Closing the Modality Gap Aligns Group-Wise Semantics
- Mitigate the Gap: Improving Cross-Modal Alignment in CLIP
- RECLIP: Resource-efficient CLIP by Training with Small Images
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- PatchDropout: Economizing Vision Transformers Using Patch Dropout
- HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment