TL;DR
PRUDENT (PRUning with DissENT) is a two-pass visual token pruning strategy that reallocates computational resources to the questions that need them most. By using a small "checker" VLM to identify when a large model's pruned answer disagrees with a full-image view, PRUDENT significantly outperforms existing one-pass pruning methods, especially at tight token budgets.
What Problem Does This Paper Address?
Large Vision-Language Models (VLMs) are computationally expensive because they process hundreds or thousands of visual tokens for every query. Visual token pruning—keeping only the most "important" tokens—is a standard way to speed them up.
However, current methods typically use one-pass pruning: they rank tokens, keep a fixed percentage (the budget), and answer the question once. The problem is that pruning damage is uneven. While most questions are easy, some rely on tiny details (like a single digit in a document) that are easily pruned away. Because the large VLM only sees the pruned image, its own confidence scores are often blind to what it has lost. The authors identify a need for a test-time mechanism to detect these "pruning failures" and fix them.
Methods
The authors propose PRUDENT, which moves away from fixed per-question budgets toward an adaptive reallocation of tokens.
- The First Pass: The large VLM answers the question using a very small token budget (e.g., half of the total average budget).
- The Checker: Simultaneously, a much smaller VLM (the "checker") answers the same question but is given the full, unpruned image.
- Dissent Detection: PRUDENT compares the two answers. If the large VLM’s pruned answer disagrees with the small VLM’s full-image answer, it’s a red flag that critical evidence might have been pruned.
- The Re-answer: Questions flagged by dissent (or low confidence) are re-answered by the large VLM using a much larger token budget (typically $2\times$ the average budget), while the "easy" questions remain at the low-budget first-pass answer.
To ensure the checker is reliable, it "abstains" from flagging if its own internal confidence is too low or if the answer is too long to match word-for-word reliably.
Datasets
The researchers evaluated PRUDENT on 18 benchmarks across three main categories:
- Text-rich: ChartQA, DocVQA, InfoVQA, TextVQA, OCRBench, and WebSRC (where fine details matter most).
- General VQA: GQA, VQAv2, OK-VQA, MMBench, MMBench-CN, MME, POPE, SEED-Bench, AI2D, and ScienceQA-IMG.
- Grounding: RefCOCO (testA and testB).
They tested these across multiple model families, including InternVL3 (8B to 78B), Qwen3-VL, and Gemma 4.
Key Insights and Ideas
- Context Trumps Scale for Details: A small model (e.g., 2B) seeing the whole image is often a better judge of truth than a giant model (e.g., 38B) seeing only 2.5% of the image. The giant model "doesn't know what it doesn't know."
- Dissent as a Signal: Disagreement between a pruned view and a full view is a much stronger predictor of pruning failure than the model's own posterior probability. As shown in Figure 3, the AUROC for detecting failures using dissent is significantly higher than using confidence alone.
- The "Detail" Bottleneck: Pruning failures are concentrated on questions requiring "small evidence." By saving tokens on easy "general" questions, PRUDENT can afford to spend $4\times$ or $8\times$ more tokens on the few questions that actually require them.
Key Results
- State-of-the-Art Performance: PRUDENT outperformed eight existing pruning methods (including IF-Prune and SGL) on every benchmark in the main tables.
- Efficiency Gains: At a 2.5% token budget, PRUDENT achieved scores that would require 3.1× to 3.5× more tokens for competing methods to match.
- Budget Robustness: The performance lead of PRUDENT actually widens as the token budget gets tighter, proving its effectiveness for extreme acceleration.
Limitations and Future Work
- Short Answer Focus: The current matching logic relies on comparing short, formatted answers. Extending this to long-form, multi-turn dialogue or open-ended generation would require more sophisticated semantic similarity checks.
- Single Image Constraint: The study focused on single-image tasks. Future work could explore how "dissent" signals work in video VLMs or multi-image reasoning.
- System Latency: While token counts are reduced, the two-pass nature adds some overhead. The authors suggest that techniques like speculative decoding can help mitigate the latency of the second pass.
Summary updated