RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding

Hengjie Zhu, Dayan Wu, Xinze Liu, Peng Fu, Zheng Lin, Weiping WangarXiv 2026 · arXiv

TL;DR

RECAP (REcycling Context-Aware Predictions) is a training-free speculative decoding framework for Vision-Language Models (VLMs) that accelerates inference by "recycling" the model's own previous predictions. By using multi-length contextual keys and fusing request-local and persistent memories, it achieves speedups of up to 1.66× without the need for an auxiliary draft model or additional training.

What Problem Does This Paper Address?

Autoregressive decoding in large models is notoriously slow because it generates text one token at a time. Speculative decoding speeds this up by using a lightweight "drafter" to propose multiple tokens that the main model verifies in parallel. However, traditional drafters require expensive training and additional GPU memory for every new model.

A cheaper alternative is prediction recycling, which stores the main model's previous outputs in a look-up table to use as future proposals. The problem? Existing methods like Token Recycling only index predictions by the last token. In multimodal settings, this leads to condition aliasing: the "correct" next token depends heavily on the specific image and the preceding text. For example, the token "her" might be followed by "head" in one image context but "feet" in another. As shown in Figure 1, simple token-based reuse fails most often at these image-sensitive decoding states.

Figure 1: The role of context in prediction recycling. Top: a cached her→head transition fails when reused after a different prefix. Middle: shorter accepted paths accompany stronger root image sensitivity (left), and source preference varies with candidate visual alignment (right). Bottom: RECAP retrieves and fuses predictions at multiple contextual resolutions.

Methods

RECAP introduces a more sophisticated way to store and retrieve recycled predictions:

  1. Context-Stratified Memory: Instead of just the last token, RECAP stores predictions under one-, two-, and three-token keys.
  2. Dual Memory Scopes:
    • Contextual Memories ($G$ and $C$): These are "request-local" and reset for every new image/prompt. This implicitly conditions the recycled predictions on the current image without needing a separate vision encoder for the drafter.
  3. Persistent Memory ($U$): A token-level map that persists across different requests to maintain high coverage for general language patterns.
  4. Entropy-Confidence Fusion: When looking up a prefix, RECAP often finds matches at multiple lengths. Rather than just picking the longest match, it fuses the distributions using weights based on specificity (favoring longer keys) and entropy-based confidence (favoring "sharper" predictions where the model was more certain).
  5. Tree Construction: These fused predictions are composed into a candidate tree using a best-first search. This is supplemented by a suffix automaton that pulls in long, contiguous sequences already seen in the prompt or history.
Figure 2: The RECAP framework. (A) Three-, two-, and one-token keys retrieve Top-K prediction rows from G, C, and U; only U survives request boundaries. (B) Entropy-confidence weighting fuses the retrieved rows under specificity priors and normalizes the retained candidates. (C) Best-first tree search repeats retrieval and fusion under each node’s path context. (D) Suffix retrieval inserts a contiguous continuation from the prompt and committed history. (E) One packed target pass commits the matching path plus one target-selected token. All evaluated nodes refresh prediction memory; only committed tokens extend sequence history. Tokens and scores are illustrative.

Datasets

The authors evaluated RECAP across several diverse VLM benchmarks:

  • Single-turn benchmarks: MMT-Bench, SEED-Bench, ScienceQA, OCRBench, ChartQA, MathVista, and TextVQA.
  • Multi-turn benchmark: MMSpec, which covers categories like reasoning, captioning, and general VQA.
  • Models: The system was tested on Qwen2.5-VL-7B and LLaVA-1.6-7B.

Key Insights and Ideas

  • The Specificity-Coverage Trade-off: Longer keys (3 tokens) are highly accurate but rarely match (low coverage). Shorter keys (1 token) match often but are prone to aliasing. RECAP's fusion mechanism dynamically balances these two.
  • Image Sensitivity: The authors found that rejected drafts in standard recycling are highly correlated with states where the model's prediction changes significantly if the image is swapped. By making the memory request-local, RECAP effectively "specializes" its proposals to the current image.
  • Training-Free Efficiency: Unlike learned drafters (e.g., EAGLE or Medusa), RECAP requires zero parameters and adds negligible overhead—occupying only about 8.7 MiB of host memory.

Key Results

RECAP consistently outperformed existing training-free baselines:

  • Speedup: Achieved a mean speedup of 1.66× on Qwen2.5-VL and 1.56× on LLaVA-1.6 across single-turn benchmarks. On the MMSpec benchmark, speedups reached up to 1.89×.
  • Acceptance Length: The average number of tokens accepted per target model pass ($\tau$) was significantly higher than the previous state-of-the-art (Token Recycling), reaching 3.02 vs 2.71 on Qwen.
  • Robustness: As shown in Figure 3, the multi-granularity approach successfully navigates the coverage-recall trade-off, with finer keys providing much higher precision (Recall@8) when they do match.
Figure 3: The specificity–coverage trade-off and contextual backoff. (a) Query coverage and conditional Recall@8 across 23,811 states from 400 inputs. Each benchmark connects U (circle), C (square), and G (diamond). Finer keys raise conditional recall while reducing coverage. (b) Paired online throughput gains of U + C backoff against U.

Limitations and Future Work

  • Scale: The study focused on 7B-parameter models. The performance on much larger models (e.g., 70B+) or smaller, faster models remains to be seen.
  • Operating Regimes: The experiments were conducted with a batch size of one. The authors note that concurrent serving and high-throughput environments are currently untested.
  • Modality: While highly effective for images, the application to video-language models—where the "context" changes dynamically over time—is a promising future direction.
  • Hyperparameters: The specificity priors and fusion weights are currently fixed; adaptive or learned weighting schemes could potentially squeeze out more performance.

Summary updated

Abstract

Recycling a target model's own predictions offers a training-free source of proposals for speculative decoding. In vision-language models, however, existing recycling indexes predictions by the last token alone, conflating predictions computed under different images and prefixes; we find that such reuse fails most often at image-sensitive decoding states. Longer keys reduce this aliasing but match fewer states. We introduce \method, which stores target predictions under one-, two-, and three-token keys: contextual memories live only for the current request and are thus implicitly conditioned on its image, while token-level memory persists across requests to retain coverage. At each lookup, entropy-confidence weighting fuses the retrieved predictions according to their key specificity and concentration. Best-first search composes the fused transitions into a candidate tree, supplemented by suffix-matched continuations, and one target pass verifies the tree under greedy decoding. Without training or an auxiliary draft model, \method\ achieves mean speedups over autoregressive decoding of $1.66\speed$ and $1.56\speed$ across seven single-turn benchmarks on Qwen2.5-VL-7B and LLaVA-1.6-7B, exceeding the strongest training-free baseline, Token Recycling ($1.51\speed$ and $1.37\speed$). Our implementation is available in https://github.com/Senmo996/RECAP-official-code.
BibTeX
@misc{zhu2026recap,
  author = {Hengjie Zhu and Dayan Wu and Xinze Liu and Peng Fu and Zheng Lin and Weiping Wang},
  title = {{RECAP: Context-Aware Prediction Recycling with Request-Local and Persistent Memories for Multimodal Speculative Decoding}},
  year = {2026},
  eprint = {2611.05894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05894},
}
Figure