TL;DR
The Fruit-Fly-Foraging Algorithm ($F^3A$) is a training-free method that slashes the inference costs of Multimodal Large Language Models (MLLMs) by pruning unnecessary visual tokens before they reach the language backbone. By treating token pruning as a dynamic, question-conditioned search rather than a static ranking problem, $F^3A$ maintains up to 93.86% of full-token performance while using only 20% of the visual tokens across model scales up to 235B parameters.
What Problem Does This Paper Address?
As vision-language models scale up, they handle high-resolution images, videos, and multi-image documents by converting them into massive sequences of visual tokens. While this improves fine-grained perception (like reading small text or analyzing charts), it introduces a massive computational bottleneck. Visual tokens dominate the "prefill" stage of inference, balloon key-value (KV) caches, and significantly increase latency.
Existing training-free pruning methods try to solve this by dropping "redundant" tokens based on one-shot heuristics, such as generic visual similarity or attention weights from early layers. The authors argue that this static approach is fundamentally flawed. An image token's value isn't absolute; it depends entirely on the question being asked. A tiny, peripheral detail might be irrelevant for a general description task but absolutely critical for a specific verification question. Furthermore, prior work lacks a systematic understanding of how token demand scales alongside model size.
Methods
The authors introduce the Fruit-Fly-Foraging Algorithm ($F^3A$), a training-free router inspired by the way fruit flies navigate odor gradients to locate food. Operating right after the vision tower and before the language model prefill, $F^3A$ frames token selection as a three-stage task-conditioned evidence search:
- Coarse Search (Building the Task Odor Field): The prompt is converted into lightweight text queries (global context, target phrases, and task templates). These queries are matched against the visual token grid using frozen, sparse random projections ("sparse sensing heads") to generate a prompt-dependent importance map called a task odor field. The image is partitioned into windows, and the highest-scoring windows are selected as initial candidates.
- Visual Lock-on (Local Exploitation): Within the selected windows, the algorithm refines local evidence. It rewards tokens with strong local support while penalizing spatial and visual redundancy (an inhibition-of-return effect) to ensure the budget isn't wasted on identical neighboring patches.
- Rescue Jump (Exploratory Recovery): To prevent missing small, peripheral objects or counter-evidence outside the primary windows, $F^3A$ reserves a portion of the token budget for a "rescue jump." It evaluates uncertainty and coverage across the entire image to salvage under-represented but task-relevant tokens.
Because $F^3A$ relies entirely on frozen weights and deterministic templates, it requires zero model training or extra LLM forward passes, preserving the original model pipeline completely.
Datasets
The authors evaluated $F^3A$ across 11 diverse multimodal benchmarks to test different dimensions of visual reasoning (e.g., localized OCR, global scene context, and spatial relations):
- Hallucination & Presence: HallusionBench, POPE
- General VQA & Understanding: MMBench-en, MMBench-CN, CCBench, MME
- Science & Diagram Reasoning: AI2D, ScienceQA-IMG
- Real-World & Grounded VQA: RealWorldQA, VSR, Visual7W
Key Insights and Ideas
- Pruning as a Search, Not a Ranking: The core conceptual shift is treating token selection as an active, query-guided navigation problem across an evidence landscape, rather than a simple top-$K$ filtering pass.
- Larger Models are Better Token Compressors: As MLLM backbones scale up (from 2B to 235B parameters), their ability to extract performance from compressed visual sequences actually improves. The accuracy curve under aggressive token compression closely mirrors the upward trend of full-token scaling.
- Scale-Dependent Token Demand: There is no single "optimal" universal retention ratio. Instead, the number of visual tokens an MLLM needs is scale-dependent and heavily shaped by how effectively the pruning mechanism preserves task-specific evidence.
Key Results
The authors conducted extensive evaluations using the Qwen3-VL family (spanning 2B dense to 235B Mixture-of-Experts models), as well as Qwen2.5-VL and InternVL3.5.
- State-of-the-Art Compression: Across 10 distinct models and three retention ratios (60%, 40%, 20%), $F^3A$ consistently achieved the highest compressed accuracy, outperforming strong baselines like FastV, VisionZip, DivPrune, and CDPruner.
- Aggressive Retention: At a strict 20% visual token retention ratio, $F^3A$ retained an average of 93.86% of full-token performance across the board.
- Fewer Tokens for High Fidelity: When targeting a near-lossless 97% of full-token performance, $F^3A$ required an average token budget of only 39.9%, compared to 50.1% for the strongest baseline.
- Efficiency Gains: On Qwen3-VL-8B at 20% retention, $F^3A$ successfully reduced end-to-end latency from 354.7 ms to 274.3 ms and slashed the KV-cache footprint from 117.7 MB down to just 29.0 MB.
Limitations and Future Work
The primary limitation of this work is its exclusive focus on visual tokens during the prefill stage. While this targets the most prominent computational bottleneck in modern MLLMs, real-world deployments also face redundancy challenges in other areas. The authors note that expanding this task-conditioned search framework to handle other modalities—such as audio streams, tool-use execution traces, and extremely long textual contexts—remains an important and open direction for future research.
Summary updated