TL;DR
DiFiT introduces a new paradigm for visual token pruning in Multimodal Large Language Models (MLLMs) by focusing on predictive fidelity—ensuring the pruned model's internal probability distributions match the full model's. By combining distribution distillation with a learnable, multi-budget scorer, DiFiT achieves significant inference speedups while drastically reducing "negative flips" in stochastic sampling and maintaining high task accuracy.
What Problem Does This Paper Address?
As MLLMs process increasingly high-resolution images and videos, the number of visual tokens explodes, leading to high memory usage and slow "prefill" latency. Pruning—simply discarding less important tokens—is a common solution.
However, the authors argue that current pruning methods are evaluated too narrowly. Most researchers only look at whether the "greedy" (most likely) answer remains correct. This paper reveals a hidden issue called predictive drift: even if the top answer stays the same, pruning often shifts the underlying probability distribution of all possible tokens.
Why does this matter?
- Sampling Failures: If you use stochastic decoding (sampling), a drifted distribution is more likely to produce a wrong answer even if the greedy answer is correct (a "negative flip").
- Confidence Calibration: Many systems use a confidence threshold to decide whether to "accept" or "reject" an AI's answer. Predictive drift breaks these thresholds, leading to more disagreements between the full and pruned models.
Methods
The authors propose DiFiT (Distribution-Fidelity-guided Token pruning), which moves beyond simple attention-based importance to a learnable selection strategy.
Key components include:
- Dual-View Distillation (DVD): This is the "fidelity" part. During training, a student branch (pruned) is forced to match the full vocabulary probability distribution of a teacher branch (unpruned) using KL divergence. This ensures the pruned model "thinks" like the full model.
- Dual-Perception Scorer (DPS): A lightweight module that ranks tokens based on both their inherent visual importance and their relevance to the text query.
- Differentiable Top-K: To make the discrete "keep or drop" decision learnable via backpropagation, the authors use a budget-constrained sigmoid mask.
- Matryoshka Budget Loss (MBL): Inspired by Matryoshka embeddings, this allows a single trained scorer to handle multiple retention ratios (e.g., 10%, 20%, and 30%) simultaneously. The rankings are nested, so tokens selected at a 10% budget are a subset of those at 20%.
Datasets
The model was trained on a 144K-sample mixture from Cambrian-737K, including data from ChartQA, OCR-VQA, and COCO.
Evaluation spanned a massive battery of benchmarks:
- General/Science: ScienceQA, AI2D, MMMU, MME.
- Text/OCR-heavy: DocVQA, TextVQA, OCRBench.
- Object Hallucination: POPE.
- Video: MVBench.
Key Insights and Ideas
- Accuracy is a Coarse Metric: The paper's most significant "aha moment" is that greedy accuracy hides internal model degradation. Figure 1 illustrates how predictive drift (KL/JS divergence) grows as compression increases, directly correlating with increased sampling errors.
- Distribution Matching > Answer Supervision: While training on reference answers (cross-entropy) helps, matching the full teacher distribution provides a much richer signal for which visual evidence is truly essential for the model's internal logic.
- One Scorer to Rule Them All: The Matryoshka approach is highly practical for engineers. You don't need to train different models for different speed/accuracy trade-offs; one scorer provides a robust ranking that works across various budgets.
Key Results
DiFiT consistently outperforms both training-free (e.g., FastV) and learnable (e.g., VisionSelector) baselines, particularly at aggressive compression levels.
- Robustness to Sampling: At 10% token retention on Qwen2.5-VL-7B, DiFiT reduced sampling-induced negative flips by 30.3% compared to FastV.
- Performance Retention: On the MME benchmark, it preserved 98.74% of the baseline performance while keeping only 30% of visual tokens.
- Efficiency: In video tasks (MVBench), DiFiT achieved a 2.34x prefill speedup and reduced peak GPU memory from 25.97 GB to 16.45 GB at 20% retention.
- Qualitative Superiority: Qualitative examples show DiFiT is better at retaining "relational" visual context—for instance, keeping the specific text on a banner along with the surrounding pixels that identify its location.
Limitations and Future Work
- Hard Selection: DiFiT uses "hard" pruning, meaning tokens are either kept or deleted. If essential information is spread thinly across many tokens, discarding them entirely might lose details that a "merging" or "pooling" approach might have saved.
- Frozen Encoder: The vision encoder remains frozen. The authors suggest that allowing the encoder to adapt its features to a smaller token budget could further improve performance.
- Generation Length: The current behavioral analysis focuses on relatively short responses. Future work could explore how predictive drift accumulates over very long sequences (e.g., multi-paragraph reasoning).
Summary updated