DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models

Jingyuan Zhang, Nihao Niu, Xinyang Lv, Beibei Han, Xiaoyun Wang, Yanning Hou, Ke XuarXiv 2026 · arXiv

TL;DR

DiFiT introduces a new paradigm for visual token pruning in Multimodal Large Language Models (MLLMs) by focusing on predictive fidelity—ensuring the pruned model's internal probability distributions match the full model's. By combining distribution distillation with a learnable, multi-budget scorer, DiFiT achieves significant inference speedups while drastically reducing "negative flips" in stochastic sampling and maintaining high task accuracy.

What Problem Does This Paper Address?

As MLLMs process increasingly high-resolution images and videos, the number of visual tokens explodes, leading to high memory usage and slow "prefill" latency. Pruning—simply discarding less important tokens—is a common solution.

However, the authors argue that current pruning methods are evaluated too narrowly. Most researchers only look at whether the "greedy" (most likely) answer remains correct. This paper reveals a hidden issue called predictive drift: even if the top answer stays the same, pruning often shifts the underlying probability distribution of all possible tokens.

Why does this matter?

  1. Sampling Failures: If you use stochastic decoding (sampling), a drifted distribution is more likely to produce a wrong answer even if the greedy answer is correct (a "negative flip").
  2. Confidence Calibration: Many systems use a confidence threshold to decide whether to "accept" or "reject" an AI's answer. Predictive drift breaks these thresholds, leading to more disagreements between the full and pruned models.

Methods

The authors propose DiFiT (Distribution-Fidelity-guided Token pruning), which moves beyond simple attention-based importance to a learnable selection strategy.

Fig. 2. Overview of DiFiT. DVD matches the softly weighted student view to full-token response distributions. DPS scores visual importance and text relevance. MBL aligns soft masks with hard selection across budgets. CE and DVD supervise the 30% student path; MBL supervises masks at 30%, 20%, and 10%. Inference gathers hard Top-K tokens into a shortened sequence for generation.

Key components include:

  • Dual-View Distillation (DVD): This is the "fidelity" part. During training, a student branch (pruned) is forced to match the full vocabulary probability distribution of a teacher branch (unpruned) using KL divergence. This ensures the pruned model "thinks" like the full model.
  • Dual-Perception Scorer (DPS): A lightweight module that ranks tokens based on both their inherent visual importance and their relevance to the text query.
  • Differentiable Top-K: To make the discrete "keep or drop" decision learnable via backpropagation, the authors use a budget-constrained sigmoid mask.
  • Matryoshka Budget Loss (MBL): Inspired by Matryoshka embeddings, this allows a single trained scorer to handle multiple retention ratios (e.g., 10%, 20%, and 30%) simultaneously. The rankings are nested, so tokens selected at a 10% budget are a subset of those at 20%.

Datasets

The model was trained on a 144K-sample mixture from Cambrian-737K, including data from ChartQA, OCR-VQA, and COCO.

Evaluation spanned a massive battery of benchmarks:

  • General/Science: ScienceQA, AI2D, MMMU, MME.
  • Text/OCR-heavy: DocVQA, TextVQA, OCRBench.
  • Object Hallucination: POPE.
  • Video: MVBench.

Key Insights and Ideas

  • Accuracy is a Coarse Metric: The paper's most significant "aha moment" is that greedy accuracy hides internal model degradation. Figure 1 illustrates how predictive drift (KL/JS divergence) grows as compression increases, directly correlating with increased sampling errors.
Fig. 1. Probability drift and behavior changes beyond greedy answers. Left: KL/JS drift on OCRBench common-correct sets at each budget, a broad control. Middle: four-task accept/reject disagreement and selective-risk increase when reusing a full-model threshold set for 90% coverage. Right: negative flips and sampled score on sets all systems answer correctly under greedy decoding (N = 7,141 at 20%; N = 4,427 at 10%). Pale-red bands identify DiFiT and are not confidence intervals. Figure 3 and Table X provide stricter output controls and adjusted drift–behavior associations.
  • Distribution Matching > Answer Supervision: While training on reference answers (cross-entropy) helps, matching the full teacher distribution provides a much richer signal for which visual evidence is truly essential for the model's internal logic.
  • One Scorer to Rule Them All: The Matryoshka approach is highly practical for engineers. You don't need to train different models for different speed/accuracy trade-offs; one scorer provides a robust ranking that works across various budgets.

Key Results

DiFiT consistently outperforms both training-free (e.g., FastV) and learnable (e.g., VisionSelector) baselines, particularly at aggressive compression levels.

  • Robustness to Sampling: At 10% token retention on Qwen2.5-VL-7B, DiFiT reduced sampling-induced negative flips by 30.3% compared to FastV.
  • Performance Retention: On the MME benchmark, it preserved 98.74% of the baseline performance while keeping only 30% of visual tokens.
  • Efficiency: In video tasks (MVBench), DiFiT achieved a 2.34x prefill speedup and reduced peak GPU memory from 25.97 GB to 16.45 GB at 20% retention.
  • Qualitative Superiority: Qualitative examples show DiFiT is better at retaining "relational" visual context—for instance, keeping the specific text on a banner along with the surrounding pixels that identify its location.

Limitations and Future Work

  • Hard Selection: DiFiT uses "hard" pruning, meaning tokens are either kept or deleted. If essential information is spread thinly across many tokens, discarding them entirely might lose details that a "merging" or "pooling" approach might have saved.
  • Frozen Encoder: The vision encoder remains frozen. The authors suggest that allowing the encoder to adapt its features to a smaller token budget could further improve performance.
  • Generation Length: The current behavioral analysis focuses on relatively short responses. Future work could explore how predictive drift accumulates over very long sequences (e.g., multi-paragraph reasoning).

Summary updated

Abstract

Visual token pruning reduces the inference cost of multimodal large language models, but answer accuracy only partly reflects its effect. Pruning can change the probabilities of alternative response tokens even when greedy answers remain correct and unchanged. We compare full and pruned response distributions under shared reference prefixes. Average predictive drift grows with compression on fixed correctly answered samples. Separate analyses associate larger drift with more confidence-decision disagreements and sampling failures. We propose DiFiT (Distribution-Fidelity-guided Token pruning) to learn discrete selection from reference answers and full-token response distributions. Dual-View Distillation matches probabilities across the vocabulary, while answer supervision supports task quality. A learned scorer and differentiable Top-$K$ learn which visual evidence to retain. A multi-budget mask objective supports different retention ratios through one shared ranking over a frozen model. On Qwen2.5-VL-7B, DiFiT reduces sampling-induced negative flips by 30.3% relative to FastV at 10% retention on four-task greedy-correct controls. It preserves 98.74% of baseline performance on MME at 30% retention.
BibTeX
@misc{zhang2026difit,
  author = {Jingyuan Zhang and Nihao Niu and Xinyang Lv and Beibei Han and Xiaoyun Wang and Yanning Hou and Ke Xu},
  title = {{DiFiT: Distribution-Fidelity-guided Visual Token Pruning for Multimodal Large Language Models}},
  year = {2026},
  eprint = {2611.15069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15069},
}
Figure