Large Vision-Language Models (VLMs) are highly capable but suffer from severe inference latency. Because they generate text autoregressively—token by token—each step requires running a massive neural network. In multimodal settings, this bottleneck is exacerbated by complex data routing and the sheer volume of visual data streams, leading to high inter-token latency and performance variability compared to text-only models, as detailed by Chitty-Venkata et al..
To accelerate these models, researchers have turned to speculative decoding. In text-only language models, speculative decoding uses a small, fast "draft" model to generate a sequence of candidate tokens, which a large "target" model then verifies in parallel in a single forward pass. If the target model accepts the draft tokens, the system skips several expensive autoregressive steps.
However, directly porting speculative decoding to VLMs introduces a fundamental cross-modal acceleration gap. Understanding why standard speculation fails in multimodal contexts—and how recent frameworks overcome these hurdles—reveals a new paradigm for efficient multimodal inference.
The Multimodal Speculation Breakdown
Standard speculative decoding fails for VLMs due to two primary challenges: visual token overhead and cross-modal misalignment.
1. The Visual Token Overhead
Images are typically processed by a vision transformer and projected into hundreds or thousands of visual tokens. When a small draft model attempts to process these visual tokens, its computational advantage evaporates. As Kang et al. observe, the overhead of processing dense visual features through the draft model often completely offsets the wall-clock time saved by parallel verification.
2. Cross-Modal Misalignment
If you try to bypass the token overhead by using a pure, text-only draft model that ignores the image entirely, a different problem arises. As noted by Ganesan et al., a text-only draft model cannot generate visually grounded predictions. Without seeing the image, the draft model's guesses diverge immediately from the target model's visually informed distribution, leading to a near-zero acceptance rate and zero net acceleration.
Standard Speculation:
[Image] ──> [Thousand Visual Tokens] ──> [Small Draft Model] ──> High Compute Overhead (Slow)
OR
[Image] ──> [Ignored by Draft] ──> [Text-Only Draft] ──> Low Acceptance Rate (No Speedup)
To make speculative decoding work for VLMs, the draft model must be given a lightweight, highly efficient way to "see" what the target model sees.
Compressing the Visual Context
To eliminate the visual token overhead, recent architectures focus on aggressive token compression specifically tailored for the draft model. The goal is to preserve the semantic essence of the image while minimizing the sequence length the draft model has to process.
In the ViSpec framework, Kang et al. introduce a lightweight vision adaptor designed for token compression alongside a global feature injection mechanism. This allows the draft model to maintain a global understanding of the visual context without choking on thousands of individual token embeddings.
Similarly, the DREAM framework proposed by Hu et al. utilizes intelligent visual data compression to ensure that the draft model remains nimble, preventing the visual context from overwhelming the draft's self-attention layers.
Aligning Draft and Target Distributions
Even with compressed visual tokens, a draft model must stay tightly synchronized with the target model's internal representations to maintain a high token acceptance rate. Researchers achieve this through feature fusion and distillation.
Cross-Attention and Hidden State Fusion
Rather than forcing the draft model to process the image independently, frameworks leverage the work already done by the target model. As outlined in a preliminary proposal by Yang et al., speculative decoding can be enhanced by feeding the draft model rich contextual information directly from the target model's cross-modal processing, including target vision embeddings, hidden states, and Key-Value (KV) caches.
The DREAM framework by Hu et al. operationalizes this intuition using an entropy-adaptive cross-attention fusion mechanism. By dynamically syncing the draft model with refined target features based on the uncertainty (entropy) of the generation, the draft model's predictions remain tightly aligned with the target model's distribution.
[Target Model Features / KV Cache]
│
▼ (Cross-Attention Fusion)
[Compressed Visual Input] ──> [Draft Model] ──> High-Quality, Visually Grounded Drafts
Self-Data Distillation
Another approach is to explicitly train the draft model to mimic the target model's multimodal behavior. In the MASSV framework, Ganesan et al. connect a smaller language model to the target VLM's vision encoder and fine-tune it using self-generated data from the target model. This self-data distillation teaches the draft model exactly how the target model translates specific visual features into text, ensuring high alignment without requiring a massive parameter footprint.
Robustness Through Diverse Drafting Strategies
Because multimodal prompts vary wildly—ranging from text-heavy documents to complex visual scenes—a single drafting strategy might not always be optimal. For instance, some images are better represented via textual descriptions (captions), while others require direct visual embeddings.
To address this, Lee et al. developed In-batch Ensemble Drafting (IbED). Instead of relying on a single draft model configuration, IbED combines multiple drafting strategies within the same batch. By ensembling different methods of incorporating visual information (such as direct token inputs versus text captions), the system achieves robust acceleration across diverse multimodal tasks, mitigating the risk of a single strategy failing and slowing down the pipeline.
Alternative Paths to Efficiency
While speculative decoding focuses on accelerating the generation phase by predicting tokens, other parallel research directions target the input length bottleneck itself. For example, in the context of multimodal in-context learning (where models are fed multiple image-text examples), input lengths explode, drastically slowing down inference.
To combat this without traditional speculation, Talemi et al. introduced Hyper-ICL, a framework that calibrates attention distributions using hyperbolic geometry to distill anchor features. This allows the model to perform in-context learning without needing explicit visual demonstrations at inference time, bypassing the latency bottleneck entirely for specific downstream tasks.
Summary
Speculative decoding for vision-language models works because it shifts the heavy lifting of multimodal processing away from raw token generation. By compressing visual tokens (Kang et al.; Hu et al.) and fusing the draft model with the target model's internal states (Hu et al.; Yang et al.), these frameworks allow a lightweight model to accurately anticipate a giant model's outputs. The result is a highly accelerated inference pipeline that retains the full accuracy and expressive power of the original VLM.
Key Foundational Work
【Proposal】Efficient Inference for Large Multimodal Models (2024) — contributed 6 concept elements to this direction.
MoE-Inference-Bench: Performance Evaluation of Mixture of Expert Large Language and Vision Models (2025) — contributed 3 concept elements to this direction.
In-batch Ensemble Drafting: Robust Speculative Decoding for LVLMs (2025) — contributed 2 concept elements to this direction.