Memory-Efficient Speculative Decoding with Quantized Draft KV Cache

Saksham Gupta, Michael LiarXiv 2026 · arXiv

TL;DR

Speculative decoding speeds up LLM inference but doubles the memory burden by requiring two separate Key-Value (KV) caches. This paper demonstrates that we can slash the draft model's cache memory by over 66% using 4-bit quantization (K4V4) with negligible impact on performance. Crucially, the authors show that the common belief that "keys need more bits than values" is largely a side effect of poor quantizer design rather than an inherent property of attention.

What Problem Does This Paper Address?

Speculative Decoding (SD) is a popular technique to reduce LLM latency. It uses a small, fast "draft" model to propose a sequence of tokens, which a larger "target" model then verifies in a single parallel pass. While this saves computation, it creates a memory bottleneck: both models must maintain their own KV caches. As context lengths grow into the tens of thousands of tokens, the memory required to store these caches can exceed the memory needed for the model weights themselves.

The authors set out to determine if we can aggressively compress the draft model's KV cache. Since the target model always has the final word on which tokens are accepted, errors in the draft cache won't break the model's logic—they only potentially slow it down by reducing the "acceptance rate" of proposed tokens.

Methods

The researchers focused on quantization, specifically reducing the precision of the draft cache from 16-bit (BF16) to 4-bit. Their approach, building on the KIVI architecture, involves a few key technical choices:

  1. Asymmetric Quantizer Geometry: They compared a naive "per-token" quantizer against a more sophisticated layout. For keys, they used per-channel quantization (grouping across tokens); for values, they used per-token quantization (grouping across channels).
  2. K4V4 Split: They settled on a balanced 4-bit key and 4-bit value allocation.
  3. Residual Window: To maintain accuracy for the most recent context, they kept the 128 most recent keys in full BF16 precision before quantizing them.
  4. Target Integrity: The target model's cache and the verification logic remain in BF16, ensuring the final output is mathematically identical to standard speculative decoding.

Datasets

The authors evaluated their method across six different target/draft model pairs (including Qwen2.5, Llama 3, OLMo 2, and SmolLM2) using several benchmarks:

  • FineWeb-Edu: For general speculative decoding performance.
  • WikiText-2: For measuring KL divergence and top-1 agreement between quantized and full-precision caches.
  • PG-19: For testing long-context performance (up to 32K tokens).
  • LongBench & Synthetic Passkey Retrieval: To test how quantization affects the model's ability to retrieve specific information from a massive context window.

Key Insights and Ideas

  • The "Key Sensitivity" Myth: Previous research often suggested that Keys (K) are more sensitive to noise than Values (V) and thus need higher bit-widths (e.g., K8V4). The authors discovered this is a "quantizer geometry" problem. When using a naive per-token quantizer, keys do look fragile. However, switching to a per-channel quantizer for keys almost entirely erases the performance gap between K8V4 and K4V8.
  • Co-design is Mandatory: You cannot pick bit-widths in a vacuum. The precision of keys and values must be co-designed with the physical layout (geometry) of the quantizer to handle outliers effectively.
  • Safe Compression: Because the draft model is just a "proposer," you can be much more aggressive with its memory than you could with a standalone model. A 1.36% relative drop in acceptance rate is a tiny price to pay for a 24% reduction in total system KV memory.

Key Results

  • Memory Savings: K4V4 reduced draft cache memory by 66.20% on average. Across the entire system (target + draft), it saved 23.95% of KV memory.
  • Minimal Performance Hit: The mean token acceptance rate only dropped from 57.93% to 57.14%.
  • Long Context Scaling: At a 32K context window for Qwen2.5-1.5B, the method saved 635.8 MiB per sequence.
  • Retrieval Accuracy: In a 16K passkey retrieval task, the grouped per-channel quantizer recovered nearly all the accuracy lost by naive quantization methods, proving that 4-bit keys are sufficient if handled correctly.

Limitations and Future Work

  • The "Software Gap": While the analytical memory savings are huge, the authors' current Triton-based prototype is actually 18% to 33% slower than the BF16 baseline. This is due to the overhead of dequantizing the cache during the attention step. Real-world speedups will require highly optimized, fused "packed-cache" kernels.
  • Peak Memory: The method currently packs the cache after a full-precision prefill, meaning it doesn't yet reduce the peak memory required during the initial prompt processing.
  • Static vs. Adaptive: The study uses a static 4-bit assignment. Future work could involve "serving-time adaptive" quantization that changes precision based on the specific layer or the complexity of the request.

Summary updated

Abstract

Speculative decoding reduces generation cost, but it stores two KV caches: one for the target model and one for the draft model. We reduce this extra memory by quantizing the draft cache. The target cache and verifier remain in BF16, so that compression cannot change the target model's final correction rule. We first study how to divide precision between keys and values. A naive per-token quantizer makes keys appear much more sensitive: splitting a 12-bit budget, K8V4 beats K4V8 in acceptance by 10.87pp. However, with grouped per-channel keys, the gap becomes -0.25pp. A controlled Qwen test confirms that changing only the key quantizer can erase or reverse the preferred bit split. Hence, following KIVI, we propose K4V4 quantization with per-channel keys and per-token values. Across six model pairs, our method reduces draft cache memory by 66.20%, and combined memory across both caches by 23.95%. Mean acceptance drops from 57.93% to 57.14%, only 1.36% relative. Overall, quantized draft caches can make speculative decoding more memory efficient; as a predictive principle for efficient AI design, key and value precision must be co-designed with the quantizer layout.
BibTeX
@misc{gupta2026memory,
  author = {Saksham Gupta and Michael Li},
  title = {{Memory-Efficient Speculative Decoding with Quantized Draft KV Cache}},
  year = {2026},
  eprint = {2611.20565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20565},
}
Figure