KV-Distill: Nearly Lossless Context Compression for Transformers

Vivek Chari, Guanghui Qin, Benjamin Van DurmeOpenReview 2025 · Submission to ICLR 2025 · Withdrawn · OpenReview

Its authors withdrew this paper, so there is no full text to read or summarize.

Abstract

Sequence-to-sequence natural language tasks often benefit greatly from long contexts, but the quadratic complexity of self-attention renders usage of long contexts non-trivial. In particular, during generation, temporary representations (stored in the KV cache) account for a large portion of GPU memory usage, and scale linearly with context length. In this work, we introduce KV-Distill, a flexible compression framework for large language models (LLMs) that distills long context KV caches into significantly shorter representations. KV-Distill can be trained as a parameter-efficient adaptor for pre-trained models, and enables the compression of arbitrary spans of a context while preserving the pre-trained model's capabilities, including instruction-tuning. We do this by treating a compressed-uncompressed cache as a student-teacher pairing and applying a KL-type divergence to match the generated outputs. Our experiments show that KV-Distill outperforms other compression techniques in worst-case extractive tasks, and approaches uncompressed performance in long context question answering and summarization. Furthermore, KV-Distill can be fine-tuned on domain-specific contexts to reduce context lengths by up 95% while preserving downstream task performance. We demonstrate the generalizability of KV-Distill across various model sizes and architectures.
BibTeX
@misc{chari2025kv,
  author = {Vivek Chari and Guanghui Qin and Benjamin Van Durme},
  title = {{KV-Distill: Nearly Lossless Context Compression for Transformers}},
  year = {2025},
  url = {https://openreview.net/forum?id=p7vJ3wsm34},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2025},
}
Figure