Disagree to Prune: Reallocating Visual Tokens Where a Small VLM Dissents

Yanchao Li, Jiaqing Xie, Ben Gao, Yusu Zhang, Beilun Wang, Yanbo Wang, Xiaohua Xu, Zhongyu Huang, Tianfan Fu, Nanqing DongarXiv 2026 · arXiv

TL;DR

PRUDENT (PRUning with DissENT) is a two-pass visual token pruning strategy that reallocates computational resources to the questions that need them most. By using a small "checker" VLM to identify when a large model's pruned answer disagrees with a full-image view, PRUDENT significantly outperforms existing one-pass pruning methods, especially at tight token budgets.

What Problem Does This Paper Address?

Large Vision-Language Models (VLMs) are computationally expensive because they process hundreds or thousands of visual tokens for every query. Visual token pruning—keeping only the most "important" tokens—is a standard way to speed them up.

However, current methods typically use one-pass pruning: they rank tokens, keep a fixed percentage (the budget), and answer the question once. The problem is that pruning damage is uneven. While most questions are easy, some rely on tiny details (like a single digit in a document) that are easily pruned away. Because the large VLM only sees the pruned image, its own confidence scores are often blind to what it has lost. The authors identify a need for a test-time mechanism to detect these "pruning failures" and fix them.

Methods

The authors propose PRUDENT, which moves away from fixed per-question budgets toward an adaptive reallocation of tokens.

  1. The First Pass: The large VLM answers the question using a very small token budget (e.g., half of the total average budget).
  2. The Checker: Simultaneously, a much smaller VLM (the "checker") answers the same question but is given the full, unpruned image.
  3. Dissent Detection: PRUDENT compares the two answers. If the large VLM’s pruned answer disagrees with the small VLM’s full-image answer, it’s a red flag that critical evidence might have been pruned.
  4. The Re-answer: Questions flagged by dissent (or low confidence) are re-answered by the large VLM using a much larger token budget (typically $2\times$ the average budget), while the "easy" questions remain at the low-budget first-pass answer.
Figure 2. PRUDENT and IF-Prune on two questions. First an- swer and checker disagree, so the large VLM re-answers at 2K. IF-Prune answers at K. Green boxes mark the evidence.

To ensure the checker is reliable, it "abstains" from flagging if its own internal confidence is too low or if the answer is too long to match word-for-word reliably.

Datasets

The researchers evaluated PRUDENT on 18 benchmarks across three main categories:

  • Text-rich: ChartQA, DocVQA, InfoVQA, TextVQA, OCRBench, and WebSRC (where fine details matter most).
  • General VQA: GQA, VQAv2, OK-VQA, MMBench, MMBench-CN, MME, POPE, SEED-Bench, AI2D, and ScienceQA-IMG.
  • Grounding: RefCOCO (testA and testB).

They tested these across multiple model families, including InternVL3 (8B to 78B), Qwen3-VL, and Gemma 4.

Key Insights and Ideas

  • Context Trumps Scale for Details: A small model (e.g., 2B) seeing the whole image is often a better judge of truth than a giant model (e.g., 38B) seeing only 2.5% of the image. The giant model "doesn't know what it doesn't know."
  • Dissent as a Signal: Disagreement between a pruned view and a full view is a much stronger predictor of pruning failure than the model's own posterior probability. As shown in Figure 3, the AUROC for detecting failures using dissent is significantly higher than using confidence alone.
  • The "Detail" Bottleneck: Pruning failures are concentrated on questions requiring "small evidence." By saving tokens on easy "general" questions, PRUDENT can afford to spend $4\times$ or $8\times$ more tokens on the few questions that actually require them.
Figure 3. Pruning failures and their detection. Failures on InternVL3-38B and 8B, (a) of IF-Prune on 6 text-rich benchmarks and (b) of each existing method on 16 benchmarks. (c) AUROC of eight signals on PRUDENT’s first pass, one point per setting (Appendix J).

Key Results

  • State-of-the-Art Performance: PRUDENT outperformed eight existing pruning methods (including IF-Prune and SGL) on every benchmark in the main tables.
  • Efficiency Gains: At a 2.5% token budget, PRUDENT achieved scores that would require 3.1× to 3.5× more tokens for competing methods to match.
  • Budget Robustness: The performance lead of PRUDENT actually widens as the token budget gets tighter, proving its effectiveness for extreme acceleration.
Figure 1. PRUDENT (Ours) and eight existing pruning meth- ods. Each curve averages the five benchmarks that the IF-Prune and SGL papers both report. The vertical axis is logarithmic in the share lost, and Appendix K plots each benchmark on a linear axis.

Limitations and Future Work

  • Short Answer Focus: The current matching logic relies on comparing short, formatted answers. Extending this to long-form, multi-turn dialogue or open-ended generation would require more sophisticated semantic similarity checks.
  • Single Image Constraint: The study focused on single-image tasks. Future work could explore how "dissent" signals work in video VLMs or multi-image reasoning.
  • System Latency: While token counts are reduced, the two-pass nature adds some overhead. The authors suggest that techniques like speculative decoding can help mitigate the latency of the second pass.

Summary updated

Abstract

Visual token pruning speeds up large vision-language models (VLMs) by keeping only a part of their visual tokens. Most existing methods improve the ranking of the tokens and then answer every question once with the same budget. With strong rankings, pruning hurts only a few questions, and signals computed on the kept tokens see little of what it removed. We find that, on most of these questions, a small VLM given the full image disagrees with the pruned answer, since it still sees the missing evidence. We propose PRUDENT, which first answers with about half the budget and spends the saved tokens on re-answering the disagreeing questions at twice the budget. We show when such re-answering pays and why the small VLM should read the full image. On 18 benchmarks, PRUDENT outperforms eight existing pruning methods at the same average budget of visual tokens. Its lead widens as the budget tightens, and holds across five InternVL sizes. PRUDENT also has the highest average in two more model families. The disagreement of the small VLM flags most pruning failures of all eight methods, and re-answering on it also improves the two strongest with their own rankings. Other checkers give similar gains, and the extra tokens go where the first pass hurts the answer. These results suggest that identifying which pruned answers are trustworthy provides an additional lever for visual token pruning beyond the ranking of tokens. Code and data are available at https://github.com/LiZaiyuan0619/PRUDENT.
BibTeX
@misc{li2026disagree,
  author = {Yanchao Li and Jiaqing Xie and Ben Gao and Yusu Zhang and Beilun Wang and Yanbo Wang and Xiaohua Xu and Zhongyu Huang and Tianfan Fu and Nanqing Dong},
  title = {{Disagree to Prune: Reallocating Visual Tokens Where a Small VLM Dissents}},
  year = {2026},
  eprint = {2611.22472},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22472},
}
Figure