Vision Transformers spend the same compute on every image patch at every layer, even though deep-layer patch features become highly redundant: many patches end up encoding nearly the same thing. Progressive token pruning removes patches ("tokens") stage by stage as the network deepens. Almost every such method decides how many tokens to keep with a fixed keep rate chosen in advance, and decides which to keep by attention score. This proposal replaces both decisions with one geometric rule. At each pruning stage, keep adding the token that is least explained by the tokens already kept, and stop when every remaining token is well explained by the kept set. Images and layers whose features span few directions are pruned hard. Cluttered, many-object images keep more tokens.
The twist that makes this more than "run a diversity selector at several layers" is where the redundancy is measured. Deep features share large common directions, such as a background, lighting, or global "image identity" component, and these dominate the spectrum. Measured in raw feature space, a flock of birds against a clear sky looks low-rank and gets over-pruned. This is the failure AgilePruner reports. We measure span coverage after removing those shared directions, following the finding that high-variance components in deep features often carry background rather than task signal. The resulting test is clear. If coverage-based stopping beats a tuned fixed schedule at matched average compute, and especially on multi-object datasets, then per-image spectral redundancy is a usable budget signal. If it does not, that is worth knowing too.
Core Claim
At a pruning stage with token matrix $X \in \mathbb{R}^{N \times D}$ ($N$ tokens, $D$ channels), we first form residual features $\tilde X$. To do this we center $X$ and project out its top $m$ principal directions, with $m$ fixed once on held-out data. We then run greedy volume maximization on the kernel
$$L = \mathrm{diag}(a)\,\tilde S\,\mathrm{diag}(a),$$
where $\tilde S$ is the cosine similarity matrix of the residual features and $a$ is each token's received attention (the class-token attention in ViTs). Greedy maximization of $\det L$ is the standard approximation to finding the most probable subset under a determinantal point process (a distribution that favors sets of mutually dissimilar items). It can be computed by incremental Cholesky in $O(NK^2)$ for $K$ kept tokens, rather than the cubic cost CDPruner faces. The marginal gain of each greedy step is the weighted squared distance of the candidate from the span of the already-kept tokens. We stop when the largest remaining gain falls below a fraction $\varepsilon$ of the first gain. A single scalar $\varepsilon$ therefore sets a per-image, per-layer budget, with no keep-rate schedule. Pruned tokens are fused into their most similar kept token.
The claim is twofold. First, at matched average FLOPs, this rule beats tuned fixed-schedule pruning, and the gap is largest where images vary most in clutter. Second, the common-mode removal ($m>0$) is what prevents the low-rank multi-object failure.
Relation to Prior Work
Three papers supply the closest mechanisms, and each leaves a specific piece undone.
CDPruner selects visual tokens for a multimodal LLM by maximizing the determinant of a kernel that combines token similarity with instruction relevance. It does this once, before the language model, with a prescribed token count, and in raw embedding space. We reuse its kernel construction, but apply it progressively inside the backbone, let the determinant's marginal gains set the count, and compute similarity after removing shared directions.
AgilePruner uses the effective rank of an image's token matrix as a scalar knob for a similarity threshold. Effective rank is the exponential of the entropy of the normalized singular values. AgilePruner names two problems directly: the low-erank many-object failure, and the "diversity trap," in which diversity-driven selection keeps background tokens and raises hallucination. These two observations constrain our design. Attention weighting in $L$ is there to avoid the diversity trap. Residual-space measurement is there to fix the low-erank failure.
PITOME protects isolated tokens during merging by using a spectral energy score on the token graph. That shares our intuition, but PITOME merges at a fixed rate.
The motivation for the residual space comes from Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection. It shows that top principal components of deep features capture dominant background variation while the useful signal sits in lower-variance components. That paper selects components with labels. We only discard a few dominant ones without supervision, so we borrow the observation but not its supervised search. Priors for Diversity in Generative Latent Variable Models supplies the general argument that a determinant-based repulsive prior prevents several units from redundantly capturing the same dominant pattern. Our tokens play the role its latent dimensions play.
Two papers constrain the proposal from the other side. Which Tokens to Use? finds plain top-K attention selection matches or beats learned scorers. Token Merging reports that constant merging schedules are near-optimal. Both are serious evidence that adaptivity may not pay, and the experiments are built to confront that evidence. ToaSt prunes FFN channels using collapsing effective rank. It is orthogonal to this work: it reduces $D$ while we reduce $N$. We treat it as a composition test, not as a competitor.
Research Program
The work has three parts.
The first is a diagnostic study that can kill the idea early. On DeiT-S, for each validation image and each of the standard pruning stages (after blocks 4, 7, and 10, following DynamicViT), we find the oracle minimal keep count. This is the smallest top-attention subset that preserves the unpruned prediction and keeps the class-token feature within a fixed cosine tolerance. We then measure how well three quantities predict that oracle count: raw effective rank, residual effective rank, and the count our $\varepsilon$-stopping rule produces. We stratify by number of labeled objects on COCO and NUS-WIDE. If none of these correlates meaningfully with the oracle count, per-image adaptivity has no signal and we stop.
The second part builds the method in two settings. The first is training-free insertion into pretrained DeiT and ViT-MAE models. The second is short fine-tuning with masked attention during training, as in DynamicViT. We sweep $\varepsilon$ to trace accuracy against average FLOPs, and we sweep $m$ from 0 to a small number of components. Variable token counts per image break batching, a known problem that ATS quantifies. We therefore report three things: FLOPs, batch-size-1 latency, and throughput under length-bucketed batching. That way the gains are not hidden behind padding.
The third part moves the rule to LLaVA-1.5-7B. There we prune visual tokens progressively across the vision encoder's last stages, with text-relevance replacing class-token attention in $a$ as in CDPruner. This setting tests whether the attention weighting avoids the diversity trap measured by CHAIR and POPE.
Baselines
The naive reference points are two. One is the unpruned model. The other is a fixed top-K attention schedule with the same three stages, its per-stage keep rates tuned by grid search to match each target FLOPs level. Which Tokens to Use? identifies this as the strongest simple competitor. We give it the same tuning budget as $\varepsilon$ and $m$ combined.
The strongest existing ViT methods are EViT, ToMe, ATS and PITOME, all run on DeiT-S at matched average FLOPs.
EViT fuses inattentive tokens into one background token. It reports DeiT-S throughput rising from 2923 to 4385 images/s with a 0.3% Top-1 drop on ImageNet-1K. ATS is the existing adaptive-count method, since it samples tokens from the attention distribution. It reports 79.7% Top-1 on DeiT-S, a 0.1% drop, with 37% fewer GFLOPs. ATS matters most here because it is the existing answer to "how many tokens per image." DynamicViT reports DeiT-S at 79.3% versus 79.8% with 37% fewer FLOPs. ToMe reports a 0.2–0.3% drop at doubled throughput on ViT-L/H. PITOME reports a 0.5% average drop on ViT-MAE-H against 2.6% for standard baselines.
All baselines are re-run from public settings on the same checkpoints, hardware and measurement code. Beating them means higher Top-1 at 37% FLOPs reduction than ATS's 79.7% on ImageNet-1K. More importantly, it means a larger margin on NABirds, COCO multi-label and NUS-WIDE. For those three datasets we found no reported result on Lacuna for any token-reduction method, so we establish all numbers ourselves.
On LLaVA-1.5-7B the reference points are CDPruner and AgilePruner. CDPruner reports 94% of original accuracy across 18 benchmarks at 95% FLOPs reduction. AgilePruner reports 96.76% relative accuracy keeping 64 of 576 tokens, and a CHAIR sentence score of 52.2 against 51.0 unpruned, with pure diversity pruning at 57.4 or higher. Beating them means matching AgilePruner's relative accuracy at the same average token count while keeping CHAIR at or below 52.2.
Three ablations isolate the key ingredients. Setting $m=0$ tests whether common-mode removal matters. Replacing the stopping rule with a fixed count, while keeping greedy volume selection, separates "which tokens" from "how many." Dropping $a$ from the kernel tests the diversity trap.
Why not just…?
Why not just tune a fixed top-K schedule? This is the real threat. Which Tokens to Use? and the ToMe schedule analysis both suggest little headroom on ImageNet, where most images contain one centered object. We expect little gain there. The proposal's bet is on per-image variance in clutter. The diagnostic correlation study and the stratified multi-object results decide it. If adaptive budgets do not beat the tuned schedule on COCO and NUS-WIDE at matched average FLOPs, we will report that adaptivity does not pay.
Why not just apply CDPruner at every stage? That would give diversity selection at a fixed count, which is exactly our "fixed count" ablation with $m=0$. CDPruner's cubic determinant cost would also bite at early stages with hundreds of tokens, whereas greedy Cholesky does not. If this ablation matches the full method, then the stopping rule and the residual space add nothing.
Why not just use AgilePruner's global effective rank to set the budget? It is a single scalar computed once, and it fails on low-rank multi-object scenes by the authors' own account. We include an erank-proportional budget, $K_l \propto \mathrm{erank}(X_l)$ in raw space, as a baseline. The flock-of-birds-style subset of COCO (many instances of one category) is where we expect it to lose.
Why not just center the features before ToMe or top-K? Centering is part of our residual step, so this is a fair and cheap competitor. We run ToMe and CDPruner-style selection on centered and residual features. If they recover most of the gain, the contribution shrinks to "measure similarity in residual space." That is a smaller but still useful finding, and we would say so.
Proposed Experiments
The decisive checks are four.
First, the oracle-count correlation on DeiT-S. We look for residual-space coverage predicting the oracle budget better than raw effective rank, with the gap concentrated in images with many same-category objects.
Second, accuracy against average FLOPs curves on ImageNet-1K, NABirds, COCO multi-label and NUS-WIDE. These curves compare the method against tuned top-K, EViT, ATS, ToMe and PITOME. They report latency and bucketed throughput alongside FLOPs, so that adaptive counts are not credited with savings the GPU never sees.
Third, the $m$ ablation on a COCO subset of images containing five or more instances of one category. This directly tests the AgilePruner failure mode.
Fourth, LLaVA-1.5-7B on GQA, TextVQA, POPE and CHAIR at 64 average visual tokens, to check that attention weighting keeps hallucination near the unpruned 51.0.
As a secondary check, we compose the method with ToaSt's channel selection on DeiT-S. This shows whether token and channel savings add up, or whether pruned token sets change which channels are redundant.
References
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- Accelerating Transformers with Spectrum-Preserving Token Merging
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- Priors for Diversity in Generative Latent Variable Models
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers
- Token Merging: Your ViT But Faster
- ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
- EViT: Expediting Vision Transformers via Token Reorganizations
- Adaptive Token Sampling for Efficient Vision Transformers