FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang, Ding WangarXiv 2026 · arXiv

TL;DR

Multimodal speculative decoding typically forces a small draft model to work with a fixed, static set of visual tokens, leading to either information starvation or unnecessary computational overhead. FOVEA solves this by caching visual features into a reusable memory and dynamically retrieving a state-conditioned, variable-sized subset of visual evidence for each decoding step. Crucially, it injects this evidence via a hidden-state correction layer, achieving up to a 2.13× end-to-end speedup while keeping the autoregressive KV cache completely intact.

What Problem Does This Paper Address?

Speculative decoding accelerates large language models by using a lightweight "draft" model to quickly guess a sequence of tokens, which a larger "target" model then verifies in parallel. In multimodal settings (Vision-Language Models or VLMs), the draft model needs visual context to make accurate guesses. Existing methods typically provide this by giving the drafter a fixed visual budget—either a static, heavily compressed version of the image or a predefined subset of visual tokens.

The authors identify two major flaws in this fixed-budget paradigm:

  1. Dynamic Visual Demand: Different decoding states require vastly different amounts of visual information. Giving the drafter too little information causes poor guesses, but giving it too much introduces noise and unnecessary computational overhead.
  2. The KV Cache Dilemma: If you try to dynamically change which visual tokens are fed to the draft model at each step, you invalidate the model's Key-Value (KV) cache. Recomputing the cache from scratch destroys the latency advantages that speculative decoding is supposed to provide.

FOVEA (Focused On-demand Visual Evidence Adaptation) is introduced to provide the draft model with exactly the visual evidence it needs, exactly when it needs it, without breaking the KV cache.

Methods

FOVEA decouples visual context from the autoregressive sequence through three main mechanisms:

  1. Image-Specific Visual Memory: The target VLM encodes the input image once at the start. FOVEA projects these features into a persistent key-value visual memory that is shared across all speculative iterations.
  2. Dynamic State-Conditioned Retrieval: For each node in the draft generation tree, the draft model uses its current hidden state and tree depth to query the visual memory. Instead of retrieving a fixed number of tokens ($K$), it uses a cumulative-mass rule to dynamically select a variable number of top-scoring visual entries based on the current state's needs.
  3. Cache-Friendly Hidden-Vector Correction: Rather than inserting the retrieved visual tokens into the text prompt (which would alter the history and invalidate the KV cache), FOVEA aggregates them into a single visual readout vector. This vector is fused with the draft hidden state via a lightweight gated residual Median Layer Perceptron (MLP) right before the language model head.
Overview of the FOVEA architecture showing visual memory construction, state-conditioned retrieval, and hidden correction.

Datasets

The authors trained and evaluated FOVEA across multiple configurations:

  • Training Data: A decontaminated version of LLaVA-v1.5-Mix-665K (51,671 samples) combined with 1,000 training samples each from six task-oriented datasets: MMT-Bench, SEED-Bench-2, ScienceQA, MathVista, OCRBench, and ChartQA.
  • Evaluation Benchmarks: 9 diverse multimodal benchmarks were used to test performance: MMT-Bench, TextVQA, MME, ChartQA, MathVista, OCRBench, ScienceQA, SEED-Bench, and MMSpec. Notably, TextVQA, MME, and MMSpec served as purely zero-shot evaluation sets.

Key Insights and Ideas

  • Less is Often More: The authors' diagnostic analysis revealed that a slim 25% visual budget was actually optimal for 56% of decoding states, whereas full visual access was rarely optimal. Packing too many visual tokens into the draft model introduces distracting context that degrades drafting quality.
  • Visual Focus Shifts Intuitively: As the model generates text, its visual demand evolves. For example, when describing an image, the retrieved visual focus shifts dynamically from an object to its attributes, and then to spatial relationships, purely driven by the evolving draft state.
Qualitative example of FOVEA's state-conditioned visual retrieval shifting focus across a decoding trajectory.
  • Modifying Representations over Context: The core "aha moment" is that you don't need to change the sequence context to change what the model "sees." Modifying the hidden representation right before token scoring achieves the same grounding effect while preserving perfect KV cache compatibility.

Key Results

FOVEA was evaluated against strong speculative decoding baselines (like EAGLE-3, Dream, and Vispec) across three target backbones: LLaVA-Next-7B, LLaVA-Next-13B, and Qwen2.5-VL-7B.

  • Substantial Speedups: FOVEA achieved the highest average end-to-end wall-clock speedups across all backbones: 1.68× on LLaVA-Next-7B, 1.66× on LLaVA-Next-13B, and 1.60× on Qwen2.5-VL-7B. It reached a peak speedup of 2.13× on the MMT-Bench dataset.
  • Higher Acceptance Rates: FOVEA consistently achieved the highest average draft acceptance lengths ($\tau$ up to 3.96 tokens per lookahead step), proving that state-conditioned visual focus directly yields higher-quality token proposals.
  • Robustness: Unlike baseline methods that occasionally slowed down below the autoregressive baseline ($<1.0\times$) on specific reasoning or text-heavy tasks, FOVEA maintained robust acceleration across all 27 backbone-benchmark combinations.

Limitations and Future Work

  • Depth Constraints: To prevent draft-side computational overhead from eclipsing verification savings, the visual correction is currently restricted to early draft-tree depths ($\ell_n \le 1$). Finding a way to scale this efficiently to deeper nodes remains an open challenge.
  • Drafter Specialization: While FOVEA avoids expensive target-model fine-tuning by using precomputed target artifacts, it still requires training a separate, backbone-specific draft model block for each new VLM architecture.

Summary updated

Abstract

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.
BibTeX
@misc{zhu2026fovea,
  author = {Hengjie Zhu and Dayan Wu and Zihao Zhang and Xinze Liu and Jingxuan Yu and Peng Fu and Zheng Lin and Weiping Wang and Ding Wang},
  title = {{FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding}},
  year = {2026},
  eprint = {2608.22883},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.22883},
}
Figure