Abstract: Divya Jyoti Bajpai, Manjesh Kumar Hanawal. Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics.
2025.
Introduction
Vision-Language Models (VLMs) like LLaVA and BLIP-2 are powerful but computationally expensive. Their autoregressive decoding process—generating text one token at a time—requires a full pass through a massive neural network for every word, creating a significant bottleneck in memory bandwidth and latency.
While Speculative Decoding (SD) is a popular solution, it typically requires storing a separate "draft" model in memory alongside the main model. Self-Speculative Decoding (SSD) attempts to solve this by using the early layers of the main model as the draft. However, in VLMs, early layers often fail to integrate complex visual and textual features, leading to poor guesses or repetitive loops that ultimately slow down the verification process. FastVLM introduces an imitation-learning framework that overcomes these hurdles, speeding up VLM inference by 1.55x to 1.85x without the memory overhead of a separate model.
Methodology
The core of FastVLM is the Imitation Network (IN), a lightweight 39M-parameter MLP inserted after an early layer. This network is specifically trained to "peek ahead" and approximate the output of the final layers, acting as a shortcut to deep-layer wisdom.
Key components of the framework include:
- Multi-level Representation Fusion: To train the IN, the authors fuse the hidden states of the last four layers of the full model, providing a rich target for the imitation network to mimic.
- Decoupled Training: A critical insight is freezing the backbone of the full model while training the IN. This prevents "objective conflict," where optimizing the early layers for drafting accidentally degrades the accuracy of the full model.
- Dynamic Drafting: FastVLM uses a dynamic approach where it guesses more tokens as the context length grows, taking advantage of the model's increasing confidence.
- KV Cache Reuse: Because the draft and verification stages share the same backbone layers, the Key-Value (KV) cache is perfectly reused, significantly reducing redundant computations compared to traditional two-model speculative decoding.
Key Results
FastVLM was evaluated across a wide range of benchmarks, including MS-COCO for image captioning, MM-Vet for complex reasoning, and VQAv2 for visual question answering. The framework achieved a 1.61x speedup on BLIP-2 (FlanT5) and up to 1.85x on LLaVA-1.5-7B.
Crucially, FastVLM maintains nearly 100% of the original model's performance. For example, on the MM-Vet benchmark, the full LLaVA-7B scored 28.0, while FastVLM scored 27.8 while being nearly twice as fast. The Imitation Network is also highly efficient, being 40% smaller than a standard LM head, making it suitable for resource-constrained devices.
Discussion and Limitations
The authors note that speculative decoding is most effective for long-form generation. For short-form tasks like Visual Question Answering (VQA), where the answer is often just one or two words, the overhead of the drafting and verification cycle can outweigh the benefits. Additionally, while the draft model is excellent at common words, it can struggle with fine-grained or rare details, which the full model catches during verification.
Future work may explore more complex fusion strategies for the imitation network and apply the framework to even larger models (e.g., 30B+ parameters) where the relative gains of self-speculative decoding could be even more significant.
Summary updated