TL;DR
Current vision token pruning methods for Multimodal Large Language Models (MLLMs) apply a single fixed strategy uniformly across all inputs, ignoring the fact that different images benefit from different pruning techniques. To exploit this hidden diversity, the authors introduce VIP-Router, a plug-and-play, ultra-lightweight module that dynamically selects the optimal pruning strategy for each sample using low-cost preview features. This sample-adaptive approach yields a 26.9% relative improvement in accuracy and a 22.0% relative increase in utility without modifying the underlying model weights.
What Problem Does This Paper Address?
MLLMs are powerful but computationally expensive. High-resolution images generate hundreds or thousands of visual tokens, which dominate memory and latency during the language model's generation phase. To mitigate this overhead, researchers have developed various vision token pruning methods (like FastV, VisionZip, PruMerge+, and DART) that drop redundant tokens based on heuristics such as attention scores or semantic similarity.
However, existing approaches suffer from a "one-size-fits-all" mentality, applying a single fixed pruning strategy uniformly to every input. The authors discover a massive gap in this paradigm: ranking pruning methods by their average benchmark performance hides significant sample-level variation. A strategy that is best on average might fail on more than a third of individual samples where an alternative, "weaker" strategy succeeds. In short, average-best does not mean sample-best.
Methods
Instead of designing yet another pruning heuristic, the authors propose VIP-Router (VIsion Pruning Router), a lightweight framework that adaptively routes each input sample to its most suitable pruning strategy at runtime.
The architecture operates in three stages:
- Low-Cost Feature Extraction: It uses frozen, lightweight preview encoders (CLIP ViT and Text) to extract global and token-level representations of the image and text query.
- Cross-Modal Interaction: A lightweight text-to-vision cross-attention layer allows the text query tokens to look at spatial visual features, creating a query-conditioned visual summary.
- Ratio-Conditioned Utility Prediction: Instead of training separate routers for every target token reduction ratio, a single two-layer MLP is explicitly conditioned on the target retention ratio ($r$ and $\log r$). It predicts a "utility score" (accuracy minus token cost) for each candidate strategy (FastV, VisionZip, PruMerge+, DART, and a "Full-token" fallback option for when pruning is too risky).
During inference, only the highest-scoring strategy is executed in a single forward pass. VIP-Router introduces merely 1.4M trainable parameters (0.017% of an 8B backbone) and leaves the underlying MLLM weights completely untouched.
Datasets
- Evaluation & Training: The authors reconstructed VTC-Bench Group A, a curated, pruning-sensitive subset derived from 8 established benchmarks: GQA, MMBench, MMBench-CN, MME, POPE, MMStar, OCRBench, and ChartQA. This group specifically isolates samples answered correctly with full tokens but incorrectly after standard image downsampling.
- Zero-Shot Transfer: To test generalization, the router was evaluated on four unseen benchmarks without any target-specific tuning: ScienceQA-IMG, MMMU, RealWorldQA, and SEED-Bench Image.
Key Insights and Ideas
- Sample-Wise Complementarity: Even the best fixed pruning strategy (VisionZip) misses over 1/3 of solvable samples that other strategies get right. Pruning heuristics are highly sensitive to the specific visual and linguistic context of the input.
- Cheap Previews are Enough: You don't need to run the heavy MLLM layers to know which pruning strategy will work. Low-cost, frozen CLIP features provide a sufficiently strong signal to predict strategy success beforehand.
- The "Full-Token" Safety Valve: Including full-token inference as an explicit routing option allows the model to dynamically bypass pruning when the risk of losing critical information is too high, maximizing overall utility.
Key Results
- Substantial Accuracy Gains: On VTC-Bench Group A, VIP-Router achieved a 26.9% relative improvement in average accuracy (jumping from 40.35% to 51.19%) over the best fixed strategy baseline.
- Improved Utility: It achieved a 22.0% relative increase in average utility after strictly accounting for the computational cost of retained tokens.
- Broad Generalization: VIP-Router consistently outperformed fixed baselines across multiple MLLM backbones (LLaVA-1.5, LLaVA-OneVision, InternVL3, Qwen2-VL) and successfully transferred zero-shot to unseen benchmarks like ScienceQA-IMG and RealWorldQA.
Limitations and Future Work
- Restricted to Group A: The empirical evaluations focus heavily on the pruning-sensitive Group A subset; its scaling behavior on completely unfiltered, general-perception benchmarks remains to be fully explored.
- Backbone Dependence: Direct zero-shot policy transfer across different MLLM architectures (without retraining the router) generally fails. The optimal sample-level pruning preferences are highly backbone-dependent, meaning a new router must be trained for each specific MLLM family.
Summary updated