MMFlash: Faster Speculative Decoding for Vision-Language Models

Sicheng Feng, Zigeng Chen, Song Wang, Gongfan Fang, Xinchao WangarXiv 2026 · arXiv

TL;DR

MMFlash is a novel speculative decoding framework designed to accelerate Vision-Language Models (VLMs) by prioritizing the learning of "visually dependent" tokens. By adaptively upweighting tokens that rely heavily on visual input and employing sparse attention for long video contexts, it achieves lossless decoding speedups of up to 3.58×, significantly outperforming existing state-of-the-art drafters.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) are powerful but notoriously slow due to their autoregressive nature—they generate text one token at a time. Speculative decoding offers a solution: a small, fast "drafter" model guesses a block of tokens, and a large "target" model verifies them in parallel.

However, existing drafters (like DFlash) were primarily built for text. When applied to multimodal tasks, their performance tanked. The authors discovered a critical bottleneck: drafters are much worse at predicting tokens that depend on an image (e.g., reading a specific number from a chart) compared to purely linguistic tokens. In fact, acceptance rates for these visually dependent tokens were found to be less than half of those for text-only tokens. MMFlash aims to bridge this "multimodal gap."

Methods

The authors introduce two primary innovations to make speculative decoding work for vision:

  1. Visually Grounded Loss Reweighting: To train a better drafter, the authors first define a Visual Dependence Score ($d_t$). This is calculated as the KL divergence between the target model's output distribution with the image versus without it. During training, MMFlash uses this score to adaptively upweight the loss for tokens that are both highly dependent on the image and currently poorly predicted by the drafter. This forces the drafter to spend its limited capacity on the hardest, most important multimodal tokens.

    Figure 2. Overview of MMFlash. (a) The target VLM is run twice on the same prefix x<t, with and without the image I, and the visual dependence score dt is the KL divergence between the two next-token distributions. Each masked position of a block keeps its base weight bk and gains a visual bonus that depends on dt. (b) The MMFlash drafter adopts a simple sparse attention pattern for video inputs, shrinking a context of more than 40K tokens to about 1.6K.
  2. Sparse Attention for Video: For long video inputs (which can exceed 40,000 tokens), standard attention is computationally prohibitive for a lightweight drafter. MMFlash employs a sparse attention pattern that retains only the most critical information: a few "attention sinks," strided visual tokens (every 32nd frame), all text tokens, and a local sliding window of the most recent 256 tokens. This shrinks the effective context from 40k+ tokens down to roughly 1.6k.

Datasets

The models were trained on a refined version of the LLaVA-OV-1.5-Instruct-Data (approx. 900K samples). For evaluation, the authors tested across a wide array of benchmarks:

  • Image Benchmarks: ChartQA, CharXiv (text/doc understanding), MMStar, MMBench (general VQA), and DynaMath, MathVista, MathVerse (STEM reasoning).
  • Video Benchmarks: LongVideoBench, MVBench, and Video-MME.

Key Insights and Ideas

  • The Multimodal Bottleneck: The "aha moment" came from analyzing where drafters fail. They found that the highest quartile of visually dependent tokens accounted for 44% of all first-rejections in a draft block. Simply training on multimodal data isn't enough; the training objective must explicitly prioritize visual grounding.
  • Confidence-Aware Learning: The loss reweighting isn't static. As the drafter gets better at predicting a visually dependent token, the "visual bonus" weight fades. This prevents the model from overfitting on visual tokens at the expense of general linguistic fluency.
  • Redundancy in Video: In long videos, the drafter's attention is highly localized. Most distant frames provide redundant information for the drafting task, justifying the massive context compression achieved by sparse attention.

Key Results

MMFlash consistently delivers superior performance across various model families (Qwen3.5, Gemma) and scales:

  • Speedup: It achieves a peak speedup of 3.58× on the Qwen3.5-9B model, outperforming the previous state-of-the-art EAGLE-3 and DFlash by significant margins.
  • Efficiency: On video benchmarks, the combination of sparse attention and fine-tuning improved speedups by up to 23.2% compared to standard DFlash.
  • Scalability: The method remains effective under high serving concurrency (up to 32 requests), making it viable for real-world deployment.
Figure 1. Decoding speedup comparison with SOTA speculative decoding methods (EAGLE-3 & DFlash) on SGLang.

Limitations and Future Work

While MMFlash makes great strides, the authors note several areas for improvement:

  • Video Complexity: Drafting for long videos remains fundamentally harder than for images, with lower overall acceptance rates.
  • Omni-modal Expansion: The current framework focuses on vision and language. Future work could extend this to "omni-modal" models that include audio, requiring a new metric for "audio dependence."
  • Frame-level Granularity: Moving from token-level to frame-level visual dependence could further optimize how drafters process video sequences.

Summary updated

Abstract

Speculative decoding accelerates generation by letting a lightweight drafter propose tokens for parallel verification by a target model. Diffusion drafters such as DFlash propose an entire token block in a single forward pass, but mainly focus on text-only tasks, leaving multimodal drafting underexplored. To study this setting, we first introduce a metric of token-level visual dependence based on the KL divergence between next-token distributions with and without visual input. We find that acceptance rates decrease from 46% in the lowest visual-dependence quartile to 22% in the highest, with this disparity persisting after multimodal training. Motivated by this observation, we propose MMFlash, a simple and effective framework that adaptively upweights visually dependent tokens that the drafter predicts poorly. For long video inputs, MMFlash further employs sparse attention to reduce redundant attention computation and improve drafting quality. Across various multimodal benchmarks, MMFlash achieves lossless decoding speedups of up to $3.58\times$, outperforming EAGLE-3 and DFlash by up to 195.4% and 17.1%, respectively.
BibTeX
@misc{feng2026mmflash,
  author = {Sicheng Feng and Zigeng Chen and Song Wang and Gongfan Fang and Xinchao Wang},
  title = {{MMFlash: Faster Speculative Decoding for Vision-Language Models}},
  year = {2026},
  eprint = {2611.18482},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18482},
}
Figure