DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models

Egor Maximov, Aleksei Merkelov, Vasiliy Alekseev, Ivan Krivorotov, Valentin Beliaev, Nikita Arsenin, Dmitry Bocharov, Aleksei GoncharovarXiv 2026 · arXiv

TL;DR

DCKV is the first end-to-end differentiable codec designed to compress the Key-Value (KV) cache of hybrid language models. By training a transform, quantizer, and conditional entropy model together against a frozen language model, it achieves massive compression—storing tokens in over 55x fewer bytes than standard bf16 while maintaining high performance on long-context benchmarks.

What Problem Does This Paper Address?

Large Language Models (LLMs) rely on a KV cache to avoid recomputing hidden states for every new token. For long prompts, this cache becomes a massive memory bottleneck, often exceeding the size of the model weights themselves. While "hybrid" models (like Qwen3.5) mitigate this by using linear attention or state-space layers for most of their architecture, they still require a full-attention KV cache for a subset of layers.

This remaining cache is the only part of the model state that grows linearly with prompt length, making it expensive to store for request reuse or to move between GPUs. Existing compression methods often focus on reducing the cache size during decoding (like token eviction) or use fixed, non-differentiable compression schemes. DCKV addresses the need for a highly efficient, learned compression method specifically for storing and transferring these prompt caches.

Methods

The authors introduce DCKV (Differentiable Codec for the KV cache). Unlike previous methods that treat compression as a post-processing step, DCKV is trained end-to-end using a rate-distortion objective.

  • The Architecture: As shown in Figure 1, the codec consists of an encoder that transforms a cache row into coefficients, a quantizer, and a decoder that restores the row. DCKV Architecture
  • Learned Transform: It uses a joint linear transform (based on PCA) for both keys and values across all layers, supplemented by a small nonlinear correction (a shallow MLP).
  • Differentiable Quantization: To allow gradients to flow through the discrete quantization step during training, the authors use "mixed quantization," which computes the bit-rate on noisy coefficients but reconstructs from rounded ones.
  • Conditional Entropy Models: Instead of assuming a static distribution, DCKV uses two conditional models to predict the mean and scale of each code:
  • Token-conditioned model: Uses the embedding of the current token.
  • Checkerboard context model: Predicts "odd" position codes based on already decoded "even" neighbors, allowing for parallel decoding in just two steps.
  • Training Objective: The model is optimized to minimize a loss function that balances the code length (rate) and the KL divergence between the original and reconstructed model outputs (distortion).
Figure 1: DCKV. Top: the cache row xt of a token is transformed, quantized and entropy-coded once after prefill, and restored once before decoding from its dequantized coefficients ˜yt = qt ⊙∆. Bottom: the conditional models tell the entropy coder the mean and the scale of every code from what the decoder already has: the prompt token and, at odd positions, the decoded codes of the two neighbouring tokens.

Datasets

  • Training: The codec was trained on a 1-billion token subset of FineWeb-Edu, using prompts of 4,096 tokens.
  • Calibration: PCA bases were initialized using documents from FineWeb and OpenR1-Math.
  • Evaluation: The authors tested DCKV on several rigorous benchmarks:
  • RULER (4K and 32K): A benchmark for long-context retrieval and reasoning.
  • LongBench: A comprehensive suite of 18 long-context tasks.
  • MMLU: For general knowledge and reasoning.
  • GPQA Diamond & IFEval: For short-prompt reasoning and instruction following.

Key Insights and Ideas

  • End-to-End is Key: By training the codec against the output of the frozen language model, the transform learns to preserve the information that actually matters for the model's predictions, rather than just minimizing reconstruction error.
  • Joint Basis Efficiency: Using a single joint basis for keys and values across all layers was found to be more efficient than maintaining separate bases, saving 7–10% in bit-rate for the same quality.
  • Context Matters: The conditional entropy models (token and neighbor conditioning) provide a significant "free" gain, removing 12–16% of the stored bytes without changing the model's output quality.
  • Hybrid Model Synergy: In hybrid models like Qwen3.5-2B, the recurrent state is fixed and relatively large (~9.8 MiB). DCKV makes the growing part of the state (the KV cache) so small that for prompts up to 11K tokens, the fixed recurrent state actually dominates the memory footprint.

Key Results

DCKV significantly outperforms the previous state-of-the-art, KVTC.

  • Compression Ratios: On Qwen3.5-2B, DCKV stores a compressed token in ~210 bytes (compared to 12,288 bytes in bf16). This is a 55x+ reduction with a performance loss of less than 1.8 points across long-context benchmarks.
  • Superiority over KVTC: At equal scores on RULER 4K, DCKV requires 72% fewer bytes than KVTC.
  • Long Context Performance: As prompt length increases, the benefits grow. At 32K tokens, DCKV scores 6.5 points higher than KVTC at a similar total request memory.
  • Visual Trade-offs: Figure 2 illustrates the clear Pareto improvement of DCKV over linear codecs and baseline methods across different bit-rates. Score change vs Payload
Figure 2: Score change against the payload (bytes per compressed token; bf16 takes 12,288). Lines connect the runs of one family. Not shown: xKV (3,151 and 3,667 bytes per token) and, in (a), the linear codec with the noise proxy at λ=3·104 (79 bytes, −25.3 points).

Limitations and Future Work

  • Model Specificity: The current results are focused on the Qwen3.5-2B hybrid model. While the principles should generalize, the codec weights are specific to the model it was trained against.
  • Decoding Memory: DCKV is a storage/transfer codec. It restores the full bf16 cache before decoding, meaning it does not reduce peak GPU memory usage during the generation process itself.
  • Rate Control: Currently, each compression rate ($\lambda$) requires its own training run. The authors experimented with "range codecs" to handle multiple rates with one model, but found they slightly underperformed single-rate versions at high compression.
  • Hardware Consistency: Ensuring that floating-point predictions in the entropy model are bit-identical across different hardware devices remains an open engineering challenge for deployment.

Summary updated

Abstract

Hybrid language models keep a key-value (KV) cache only in their few full-attention layers, but this cache is still the part of a prompt's state that grows with the prompt and that must be stored or moved to reuse it. We present DCKV, a differentiable codec for the KV cache and the first KV-cache codec that is trained end to end. Its transform, its quantizer and its entropy models, which are conditioned on the prompt tokens and on already decoded neighbouring positions, are optimized together against the frozen language model, both for the code length that the entropy models assign and for the change of the model's output. On Qwen3.5-2B, DCKV stores a compressed token in more than 55 times fewer bytes than bf16 at a loss of at most 1.8 points on three long-context benchmarks, and in more than 100 times fewer bytes at a loss of at most 3.6 points.
BibTeX
@misc{maximov2026dckv,
  author = {Egor Maximov and Aleksei Merkelov and Vasiliy Alekseev and Ivan Krivorotov and Valentin Beliaev and Nikita Arsenin and Dmitry Bocharov and Aleksei Goncharov},
  title = {{DCKV: Learned Transform Coding of the Prompt KV Cache in Hybrid Language Models}},
  year = {2026},
  eprint = {2610.36950},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.36950},
}
Figure