TL;DR
Multimodal speculative decoding typically forces a small draft model to work with a fixed, static set of visual tokens, leading to either information starvation or unnecessary computational overhead. FOVEA solves this by caching visual features into a reusable memory and dynamically retrieving a state-conditioned, variable-sized subset of visual evidence for each decoding step. Crucially, it injects this evidence via a hidden-state correction layer, achieving up to a 2.13× end-to-end speedup while keeping the autoregressive KV cache completely intact.
What Problem Does This Paper Address?
Speculative decoding accelerates large language models by using a lightweight "draft" model to quickly guess a sequence of tokens, which a larger "target" model then verifies in parallel. In multimodal settings (Vision-Language Models or VLMs), the draft model needs visual context to make accurate guesses. Existing methods typically provide this by giving the drafter a fixed visual budget—either a static, heavily compressed version of the image or a predefined subset of visual tokens.
The authors identify two major flaws in this fixed-budget paradigm:
- Dynamic Visual Demand: Different decoding states require vastly different amounts of visual information. Giving the drafter too little information causes poor guesses, but giving it too much introduces noise and unnecessary computational overhead.
- The KV Cache Dilemma: If you try to dynamically change which visual tokens are fed to the draft model at each step, you invalidate the model's Key-Value (KV) cache. Recomputing the cache from scratch destroys the latency advantages that speculative decoding is supposed to provide.
FOVEA (Focused On-demand Visual Evidence Adaptation) is introduced to provide the draft model with exactly the visual evidence it needs, exactly when it needs it, without breaking the KV cache.
Methods
FOVEA decouples visual context from the autoregressive sequence through three main mechanisms:
- Image-Specific Visual Memory: The target VLM encodes the input image once at the start. FOVEA projects these features into a persistent key-value visual memory that is shared across all speculative iterations.
- Dynamic State-Conditioned Retrieval: For each node in the draft generation tree, the draft model uses its current hidden state and tree depth to query the visual memory. Instead of retrieving a fixed number of tokens ($K$), it uses a cumulative-mass rule to dynamically select a variable number of top-scoring visual entries based on the current state's needs.
- Cache-Friendly Hidden-Vector Correction: Rather than inserting the retrieved visual tokens into the text prompt (which would alter the history and invalidate the KV cache), FOVEA aggregates them into a single visual readout vector. This vector is fused with the draft hidden state via a lightweight gated residual Median Layer Perceptron (MLP) right before the language model head.
Datasets
The authors trained and evaluated FOVEA across multiple configurations:
- Training Data: A decontaminated version of LLaVA-v1.5-Mix-665K (51,671 samples) combined with 1,000 training samples each from six task-oriented datasets: MMT-Bench, SEED-Bench-2, ScienceQA, MathVista, OCRBench, and ChartQA.
- Evaluation Benchmarks: 9 diverse multimodal benchmarks were used to test performance: MMT-Bench, TextVQA, MME, ChartQA, MathVista, OCRBench, ScienceQA, SEED-Bench, and MMSpec. Notably, TextVQA, MME, and MMSpec served as purely zero-shot evaluation sets.
Key Insights and Ideas
- Less is Often More: The authors' diagnostic analysis revealed that a slim 25% visual budget was actually optimal for 56% of decoding states, whereas full visual access was rarely optimal. Packing too many visual tokens into the draft model introduces distracting context that degrades drafting quality.
- Visual Focus Shifts Intuitively: As the model generates text, its visual demand evolves. For example, when describing an image, the retrieved visual focus shifts dynamically from an object to its attributes, and then to spatial relationships, purely driven by the evolving draft state.
- Modifying Representations over Context: The core "aha moment" is that you don't need to change the sequence context to change what the model "sees." Modifying the hidden representation right before token scoring achieves the same grounding effect while preserving perfect KV cache compatibility.
Key Results
FOVEA was evaluated against strong speculative decoding baselines (like EAGLE-3, Dream, and Vispec) across three target backbones: LLaVA-Next-7B, LLaVA-Next-13B, and Qwen2.5-VL-7B.
- Substantial Speedups: FOVEA achieved the highest average end-to-end wall-clock speedups across all backbones: 1.68× on LLaVA-Next-7B, 1.66× on LLaVA-Next-13B, and 1.60× on Qwen2.5-VL-7B. It reached a peak speedup of 2.13× on the MMT-Bench dataset.
- Higher Acceptance Rates: FOVEA consistently achieved the highest average draft acceptance lengths ($\tau$ up to 3.96 tokens per lookahead step), proving that state-conditioned visual focus directly yields higher-quality token proposals.
- Robustness: Unlike baseline methods that occasionally slowed down below the autoregressive baseline ($<1.0\times$) on specific reasoning or text-heavy tasks, FOVEA maintained robust acceleration across all 27 backbone-benchmark combinations.
Limitations and Future Work
- Depth Constraints: To prevent draft-side computational overhead from eclipsing verification savings, the visual correction is currently restricted to early draft-tree depths ($\ell_n \le 1$). Finding a way to scale this efficiently to deeper nodes remains an open challenge.
- Drafter Specialization: While FOVEA avoids expensive target-model fine-tuning by using precomputed target artifacts, it still requires training a separate, backbone-specific draft model block for each new VLM architecture.
Summary updated