Vision-language models spend most of their compute on image tokens, the small image patches the model reads as a sequence. A family of methods called progressive token pruning deletes the patches that look unimportant as the image passes through the network, and they cut compute by 70–90% while keeping about 96–99% of benchmark accuracy. Those benchmarks, however, are ones where test images look statistically like training images. Separately, work on compositional generalization, meaning the ability to handle new combinations of familiar parts such as a red cube when cubes were never red in training, shows that these same models lean heavily on familiar pairings. Most pruners decide what to keep from the model's own attention. So the concern is straightforward. If attention is drawn to familiar combinations, a pruner may selectively discard the patches that carry the novel combination, which is exactly the evidence a compositional test depends on. Standard benchmarks would never reveal this.
This note proposes to measure that effect directly and to separate it from its confounds. The central tool is paired counterfactual scenes. We render the same image twice and change only whether one object's attribute–shape pairing was seen in training. We then ask whether each pruner keeps fewer of that object's patches when its pairing is unfamiliar, and whether the accuracy lost to pruning is larger on compositional test splits than on ordinary ones. If the effect is real, we test a simple, training-free fix: give every candidate object region a guaranteed minimum share of the kept tokens.
Core Claim
We claim that pruning methods that rank patches by attention pay a compositional pruning tax. At a matched token budget, they lose more accuracy, relative to the unpruned model, on test examples with unseen attribute–object bindings than on examples with seen ones. We further claim the mechanism is lower survival of the novel object's patches, not a general loss of capacity. Operationally, for a pruner $P$ at keep ratio $r$, let $\rho_{\text{seen}}(P,r)$ and $\rho_{\text{unseen}}(P,r)$ be pruned accuracy divided by unpruned accuracy on the seen-binding and unseen-binding subsets. The tax is
$$T(P,r) = \rho_{\text{seen}}(P,r) - \rho_{\text{unseen}}(P,r).$$
The mechanistic counterpart is the survival gap $S(P,r)$. This is the fraction of a target object's ground-truth patches that survive pruning when its binding is familiar, minus the fraction that survive when the same object, in the same position and at the same size, carries an unfamiliar binding.
No paper in our search evaluates any token pruner on a compositional split. What would be new is the controlled, patch-level causal measurement and the resulting design constraint for pruners, not another pruner.
Motivating and Constraining Prior Work
Three lines of work motivate the hypothesis. CLIP Models Generalize Less Than Compositional Benchmarks Suggest splits binding benchmarks into seen, mixed and unseen bindings. It finds that CLIP retrieval drops by 38–43 points on unseen bindings in PUG:SPARE, and that larger models pay a bigger "familiarity tax." It studies unpruned models only; we ask whether pruning amplifies that tax. Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks shows that recall on novel object pairs tracks the log geometric-mean frequency of the parts. This suggests the rarer constituent is where pruning damage should concentrate. CLEVR supplies CoGenT, where accuracy falls from about 85% to about 51% when attribute–shape pairings are swapped. That drop gives us a split with exactly controlled bindings and rendered ground-truth object locations.
The pruning literature both supplies the methods and constrains the design. Which Tokens to Use? shows that plain top-K attention pruning matches or beats learned pruners on classification, and that light pruning even improves accuracy slightly. So top-K is the default we must beat, not a straw man. Adaptive Token Sampling and DynamicViT give the vision-encoder pruners we apply to CLIP: one samples tokens without extra parameters, and the other trains small keep/drop predictors. D2Pruner shows that multimodal models attend by grid position regardless of content, and it corrects for this by subtracting a positional prior. This is a confound we must control, because a "novel" object placed in an unfavored row would lose patches for reasons unrelated to familiarity. AgilePruner switches between attention-based and diversity-based selection using the effective rank of the embeddings. It admits failure on low-rank scenes with many small objects, which describes CLEVR-like scenes. It also documents that diversity-only selection, as in DivPrune, raises hallucination (CHAIR $C_S$ 57.4+ versus 51.0 unpruned). This rules out "just maximize diversity" as a free fix. CROP keeps a single question-relevant box, so by construction it cannot serve relational questions about two separated objects. We include it as an expected failure case.
Finally, Are Object-Centric Representations Better At Compositional Generalization? shows that compressing patches into object slots beats dense encoders by up to 12.3 points on hard compositional splits of CLEVRTex, Super-CLEVR and MOVi-C, and that simple k-means grouping of patches is competitive. This is the strongest hint that per-object structure, rather than per-patch importance, is what compositional reasoning needs. It motivates our proposed fix.
Research Program
The work has three parts, and each later part depends on the earlier one showing a real effect.
First, a behavioral audit. We apply existing pruners at keep ratios of 50%, 25% and 11% (64 of 576 tokens) to two model regimes. The first is CLIP ViT encoders evaluated on PUG:SPARE and the ARO VG-A seen/unseen subsets produced by BindSplit. There, top-K, ATS and a retrained DynamicViT operate inside the vision encoder. The second is LLaVA-1.5-7B and InternVL-2.5-8B, where D2Pruner, AgilePruner, DivPrune, CROP-ILP and top-K operate on image tokens in the language decoder. Evaluation uses CLEVR CoGenT after LoRA fine-tuning on Condition A (tested on A-val versus B-val) and C-VQA versus standard VQA v1.0. For each pruner we compute $T(P,r)$.
Second, the causal counterfactual. We render paired CLEVR/CoGenT scenes, as in Blender following the CLEVR generator, in which exactly one object's color is changed between a Condition-A-familiar pairing and a Condition-B-unfamiliar pairing. Geometry, lighting and every other object are held fixed. Because renders give exact object masks, we compute $S(P,r)$ per object and per layer. To separate familiarity from position, we fully cross target location over the 3×3 image regions and apply D2Pruner's positional prior as a covariate. A matching counterfactual on PUG:SPARE swaps the color of an animal. If familiar and unfamiliar variants survive at equal rates, Gap 1 is false, and the paper reports that pruners are compositionally safe. That would itself be a useful negative result.
Third, the fix, pursued only if $S$ is substantially positive. Segment-quota pruning groups patches into candidate regions with k-means over early-layer features plus 2D coordinates, following the k-means grouping in the object-centric study. It reserves a minimum number of kept tokens per region, then fills the remaining budget by debiased attention within and across regions. The fix needs no training and costs one small clustering step. Its purpose is narrow: to test whether guaranteeing each object a presence removes the tax, not to set new records on general VQA.
Baselines
The naive reference is the unpruned model at full token count. It defines the ratios $\rho$ and is the ceiling for every method. The second reference is uniform spatial downsampling, which pools the grid 2× or 3× per side to match the keep ratio. It ignores content and so cannot be biased by familiarity. If it shows zero tax while attention pruners show a positive tax, that directly supports the mechanism.
The strongest existing competitor for this question is D2Pruner. It already removes positional bias and enforces spatial coverage through a neighborhood graph, so it is the method most likely to avoid the tax without any new idea. On LLaVA-1.5-7B it reports 99.2% of unpruned performance at a 74.2% FLOP reduction, and 85.7% on RefCOCO at 90% reduction on InternVL-2.5-8B (D2Pruner). AgilePruner reports 96.76% relative performance at 64 tokens on LLaVA-1.5-7B (AgilePruner), and CROP-ILP reports 96.0–96.2% at 88.9% pruning (CROP). For any of these pruners on CoGenT, C-VQA, PUG:SPARE or BindSplit subsets, no reported result was found in our search. For the encoder regime, top-K attention is the strongest parameter-free competitor (Which Tokens to Use?).
For fairness, every pruner sees the same fine-tuned checkpoint, the same images and prompts, and exactly matched token counts at each layer. We report matched prefill FLOPs, which include clustering overhead for our method and the stated 0.8% for D2Pruner and 3.2% for AgilePruner. Each method gets the same small hyperparameter search over its own knobs, tuned only on Condition-A or seen-binding validation data, never on unseen splits.
Beating the strongest baseline means one of two outcomes at the same keep ratio. Either the tax $T$ shrinks by at least half relative to D2Pruner and does not overlap across seeds and bootstrap intervals, while seen-split accuracy stays within one point of it. Or equal tax is reached at a lower budget. Two ablations isolate the key ingredient. The first is oracle quotas that use ground-truth masks, which sets an upper bound for the quota idea. The second is quotas over random grid regions of matched count, which tests whether it is the object structure or merely spatial spreading that matters.
Why not just…?
Why not just keep more tokens? If the tax exists only at extreme budgets, the practical advice would be "don't prune below 25%," and no new method is needed. The audit sweeps keep ratios to settle this. We report the smallest budget at which $T$ becomes indistinguishable from zero for each pruner. If that budget is similar for all methods and still gives useful savings, we will say so and not push the fix.
Why not just use D2Pruner, which already debiases position and spreads tokens spatially? This is the most serious threat. Its neighborhood graph may already guarantee each CLEVR object a few surviving patches. Two considerations cut the other way, though both are untested. Its debiasing prior is estimated from generic COCO prompts and removes positional, not content, bias. Its graph also models local adjacency rather than object membership. The counterfactual survival gap $S$ for D2Pruner decides the question directly. If $S \approx 0$ and $T \approx 0$ for D2Pruner, the paper's conclusion becomes a finding that debiased, structured selection is compositionally safe while attention and single-region methods are not, and the segment-quota method is dropped.
Why not blame the model rather than the pruner? CoGenT-B accuracy is already low without any pruning, so extra loss on unseen splits could simply reflect harder examples that degrade faster under any perturbation. Two controls address this. Uniform downsampling perturbs content without regard to familiarity, so it measures the "harder examples degrade faster" baseline. The per-object survival gap is measured on identical geometry, so it does not depend on example difficulty at all.
Why not just use an object-centric encoder? The object-centric study shows slot models generalize better. However, they require dedicated reconstruction pre-training and cannot be dropped into LLaVA or InternVL. As a secondary check, we run top-K pruning and segment-quota pruning on DINOv2 features in its CLEVRTex and Super-CLEVR benchmark, at matched downstream compute. This tells us how much of the 12.3-point slot advantage per-object token quotas recover.
Proposed Experiments
The first decisive check is whether any attention-based pruner shows $T > 0$ on CoGenT-B versus A and on unseen versus seen PUG:SPARE bindings, with bootstrap confidence intervals over three fine-tuning seeds. The second is whether the counterfactual renders show $S > 0$, which would mean fewer surviving patches for the same object in the same place when only its binding is unfamiliar, after conditioning on position. The third is whether the effect is dose-dependent in real data, testing whether the survival of patches belonging to the rarer constituent tracks log pretraining frequency, as Wiedemer et al.'s frequency result predicts. The fourth, conditional on the first three, is whether oracle quotas eliminate the tax and unsupervised segment quotas recover most of that gain over D2Pruner. This check also verifies with CHAIR and POPE that the quotas do not reintroduce the hallucination increase AgilePruner reports for diversity-heavy selection. We will check CROP-ILP separately on CLEVR's two-object relational questions, where we expect it to fail, to confirm that single-region selection is incompatible with relational compositional queries.
References
- CLIP Models Generalize Less Than Compositional Benchmarks Suggest
- Pretraining Frequency Predicts Compositional Generalization of CLIP on Real-World Tasks
- CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- Adaptive Token Sampling for Efficient Vision Transformers
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
- D²Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models
- CROP: Contextual Region-Oriented Visual Token Pruning
- Are Object-Centric Representations Better at Compositional Generalization?