How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

YiJie Huang, Yiqun Zhang, Zhuoyue Jia, Xiaocui Yang, Junzhao Huang, Zihan Wang, Shi Feng, Daling Wang, Yifei Zhang, Yongkang LiuarXiv 2026 · arXiv

TL;DR

The Fruit-Fly-Foraging Algorithm ($F^3A$) is a training-free method that slashes the inference costs of Multimodal Large Language Models (MLLMs) by pruning unnecessary visual tokens before they reach the language backbone. By treating token pruning as a dynamic, question-conditioned search rather than a static ranking problem, $F^3A$ maintains up to 93.86% of full-token performance while using only 20% of the visual tokens across model scales up to 235B parameters.

What Problem Does This Paper Address?

As vision-language models scale up, they handle high-resolution images, videos, and multi-image documents by converting them into massive sequences of visual tokens. While this improves fine-grained perception (like reading small text or analyzing charts), it introduces a massive computational bottleneck. Visual tokens dominate the "prefill" stage of inference, balloon key-value (KV) caches, and significantly increase latency.

Existing training-free pruning methods try to solve this by dropping "redundant" tokens based on one-shot heuristics, such as generic visual similarity or attention weights from early layers. The authors argue that this static approach is fundamentally flawed. An image token's value isn't absolute; it depends entirely on the question being asked. A tiny, peripheral detail might be irrelevant for a general description task but absolutely critical for a specific verification question. Furthermore, prior work lacks a systematic understanding of how token demand scales alongside model size.

Methods

The authors introduce the Fruit-Fly-Foraging Algorithm ($F^3A$), a training-free router inspired by the way fruit flies navigate odor gradients to locate food. Operating right after the vision tower and before the language model prefill, $F^3A$ frames token selection as a three-stage task-conditioned evidence search:

  1. Coarse Search (Building the Task Odor Field): The prompt is converted into lightweight text queries (global context, target phrases, and task templates). These queries are matched against the visual token grid using frozen, sparse random projections ("sparse sensing heads") to generate a prompt-dependent importance map called a task odor field. The image is partitioned into windows, and the highest-scoring windows are selected as initial candidates.
  2. Visual Lock-on (Local Exploitation): Within the selected windows, the algorithm refines local evidence. It rewards tokens with strong local support while penalizing spatial and visual redundancy (an inhibition-of-return effect) to ensure the budget isn't wasted on identical neighboring patches.
  3. Rescue Jump (Exploratory Recovery): To prevent missing small, peripheral objects or counter-evidence outside the primary windows, $F^3A$ reserves a portion of the token budget for a "rescue jump." It evaluates uncertainty and coverage across the entire image to salvage under-represented but task-relevant tokens.

Because $F^3A$ relies entirely on frozen weights and deterministic templates, it requires zero model training or extra LLM forward passes, preserving the original model pipeline completely.

Figure 2: Overview of F3A. Prompt-conditioned cues guide a three-stage foraging process: coarse search, visual lock-on, and rescue jump. The selected visual tokens replace the full visual block before frozen LLM prefill, without finetuning or decoding changes.

Datasets

The authors evaluated $F^3A$ across 11 diverse multimodal benchmarks to test different dimensions of visual reasoning (e.g., localized OCR, global scene context, and spatial relations):

  • Hallucination & Presence: HallusionBench, POPE
  • General VQA & Understanding: MMBench-en, MMBench-CN, CCBench, MME
  • Science & Diagram Reasoning: AI2D, ScienceQA-IMG
  • Real-World & Grounded VQA: RealWorldQA, VSR, Visual7W

Key Insights and Ideas

  • Pruning as a Search, Not a Ranking: The core conceptual shift is treating token selection as an active, query-guided navigation problem across an evidence landscape, rather than a simple top-$K$ filtering pass.
  • Larger Models are Better Token Compressors: As MLLM backbones scale up (from 2B to 235B parameters), their ability to extract performance from compressed visual sequences actually improves. The accuracy curve under aggressive token compression closely mirrors the upward trend of full-token scaling.
  • Scale-Dependent Token Demand: There is no single "optimal" universal retention ratio. Instead, the number of visual tokens an MLLM needs is scale-dependent and heavily shaped by how effectively the pruning mechanism preserves task-specific evidence.

Key Results

The authors conducted extensive evaluations using the Qwen3-VL family (spanning 2B dense to 235B Mixture-of-Experts models), as well as Qwen2.5-VL and InternVL3.5.

  • State-of-the-Art Compression: Across 10 distinct models and three retention ratios (60%, 40%, 20%), $F^3A$ consistently achieved the highest compressed accuracy, outperforming strong baselines like FastV, VisionZip, DivPrune, and CDPruner.
  • Aggressive Retention: At a strict 20% visual token retention ratio, $F^3A$ retained an average of 93.86% of full-token performance across the board.
  • Fewer Tokens for High Fidelity: When targeting a near-lossless 97% of full-token performance, $F^3A$ required an average token budget of only 39.9%, compared to 50.1% for the strongest baseline.
  • Efficiency Gains: On Qwen3-VL-8B at 20% retention, $F^3A$ successfully reduced end-to-end latency from 354.7 ms to 274.3 ms and slashed the KV-cache footprint from 117.7 MB down to just 29.0 MB.
Figure 3: Compression-aware scaling on Qwen3-VL. (a) Average accuracy of full-token inference and F3A at 60%, 40%, and 20% retention across six scales. (b) Accuracy gain of F3A over the strongest baseline at each scale and budget.

Limitations and Future Work

The primary limitation of this work is its exclusive focus on visual tokens during the prefill stage. While this targets the most prominent computational bottleneck in modern MLLMs, real-world deployments also face redundancy challenges in other areas. The authors note that expanding this task-conditioned search framework to handle other modalities—such as audio streams, tool-use execution traces, and extremely long textual contexts—remains an important and open direction for future research.

Summary updated

Abstract

Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cost raises a basic scaling question: as multimodal models grow, how many visual tokens are actually needed, and how should they be allocated under a fixed visual token budget? Existing training-free pruning methods typically answer this with one-shot proxies such as decoder attention, visual similarity, or conditional diversity. We argue that visual token pruning is better viewed as task-conditioned evidence search, especially under aggressive compression and across model scales. We propose F^3A, a training-free router for visual token pruning that operates before the language model consumes image tokens. F^3A builds lightweight question-conditioned cues, matches them to visual-grid tokens through frozen sparse sensing heads, and allocates a fixed vision token budget via coarse evidence localization, local refinement, coverage-preserving competition, and recovery of under-covered regions. It requires no model training, no extra LLM forward pass and preserves the original multimodal prompting and decoding pipeline.
BibTeX
@misc{huang2026how,
  author = {YiJie Huang and Yiqun Zhang and Zhuoyue Jia and Xiaocui Yang and Junzhao Huang and Zihan Wang and Shi Feng and Daling Wang and Yifei Zhang and Yongkang Liu},
  title = {{How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A}},
  year = {2026},
  eprint = {2605.16359},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.16359},
}
Figure