TL;DR
RECAP (REcycling Context-Aware Predictions) is a training-free speculative decoding framework for Vision-Language Models (VLMs) that accelerates inference by "recycling" the model's own previous predictions. By using multi-length contextual keys and fusing request-local and persistent memories, it achieves speedups of up to 1.66× without the need for an auxiliary draft model or additional training.
What Problem Does This Paper Address?
Autoregressive decoding in large models is notoriously slow because it generates text one token at a time. Speculative decoding speeds this up by using a lightweight "drafter" to propose multiple tokens that the main model verifies in parallel. However, traditional drafters require expensive training and additional GPU memory for every new model.
A cheaper alternative is prediction recycling, which stores the main model's previous outputs in a look-up table to use as future proposals. The problem? Existing methods like Token Recycling only index predictions by the last token. In multimodal settings, this leads to condition aliasing: the "correct" next token depends heavily on the specific image and the preceding text. For example, the token "her" might be followed by "head" in one image context but "feet" in another. As shown in Figure 1, simple token-based reuse fails most often at these image-sensitive decoding states.
Methods
RECAP introduces a more sophisticated way to store and retrieve recycled predictions:
- Context-Stratified Memory: Instead of just the last token, RECAP stores predictions under one-, two-, and three-token keys.
- Dual Memory Scopes:
- Contextual Memories ($G$ and $C$): These are "request-local" and reset for every new image/prompt. This implicitly conditions the recycled predictions on the current image without needing a separate vision encoder for the drafter.
- Persistent Memory ($U$): A token-level map that persists across different requests to maintain high coverage for general language patterns.
- Entropy-Confidence Fusion: When looking up a prefix, RECAP often finds matches at multiple lengths. Rather than just picking the longest match, it fuses the distributions using weights based on specificity (favoring longer keys) and entropy-based confidence (favoring "sharper" predictions where the model was more certain).
- Tree Construction: These fused predictions are composed into a candidate tree using a best-first search. This is supplemented by a suffix automaton that pulls in long, contiguous sequences already seen in the prompt or history.
Datasets
The authors evaluated RECAP across several diverse VLM benchmarks:
- Single-turn benchmarks: MMT-Bench, SEED-Bench, ScienceQA, OCRBench, ChartQA, MathVista, and TextVQA.
- Multi-turn benchmark: MMSpec, which covers categories like reasoning, captioning, and general VQA.
- Models: The system was tested on Qwen2.5-VL-7B and LLaVA-1.6-7B.
Key Insights and Ideas
- The Specificity-Coverage Trade-off: Longer keys (3 tokens) are highly accurate but rarely match (low coverage). Shorter keys (1 token) match often but are prone to aliasing. RECAP's fusion mechanism dynamically balances these two.
- Image Sensitivity: The authors found that rejected drafts in standard recycling are highly correlated with states where the model's prediction changes significantly if the image is swapped. By making the memory request-local, RECAP effectively "specializes" its proposals to the current image.
- Training-Free Efficiency: Unlike learned drafters (e.g., EAGLE or Medusa), RECAP requires zero parameters and adds negligible overhead—occupying only about 8.7 MiB of host memory.
Key Results
RECAP consistently outperformed existing training-free baselines:
- Speedup: Achieved a mean speedup of 1.66× on Qwen2.5-VL and 1.56× on LLaVA-1.6 across single-turn benchmarks. On the MMSpec benchmark, speedups reached up to 1.89×.
- Acceptance Length: The average number of tokens accepted per target model pass ($\tau$) was significantly higher than the previous state-of-the-art (Token Recycling), reaching 3.02 vs 2.71 on Qwen.
- Robustness: As shown in Figure 3, the multi-granularity approach successfully navigates the coverage-recall trade-off, with finer keys providing much higher precision (Recall@8) when they do match.
Limitations and Future Work
- Scale: The study focused on 7B-parameter models. The performance on much larger models (e.g., 70B+) or smaller, faster models remains to be seen.
- Operating Regimes: The experiments were conducted with a batch size of one. The authors note that concurrent serving and high-throughput environments are currently untested.
- Modality: While highly effective for images, the application to video-language models—where the "context" changes dynamically over time—is a promising future direction.
- Hyperparameters: The specificity priors and fusion weights are currently fixed; adaptive or learned weighting schemes could potentially squeeze out more performance.
Summary updated