KV-Distill: Nearly Lossless Context Compression for Transformers
Its authors withdrew this paper, so there is no full text to read or summarize.
Lists
Only you see this
Abstract
Sequence-to-sequence natural language tasks often benefit greatly from long contexts, but the quadratic complexity of self-attention renders usage of long contexts non-trivial. In particular, during generation, temporary representations (stored in the KV cache) account for a large portion of GPU memory usage, and scale linearly with context length. In this work, we introduce KV-Distill, a flexible compression framework for large language models (LLMs) that distills long context KV caches into significantly shorter representations. KV-Distill can be trained as a parameter-efficient adaptor for pre-trained models, and enables the compression of arbitrary spans of a context while preserving the pre-trained model's capabilities, including instruction-tuning. We do this by treating a compressed-uncompressed cache as a student-teacher pairing and applying a KL-type divergence to match the generated outputs. Our experiments show that KV-Distill outperforms other compression techniques in worst-case extractive tasks, and approaches uncompressed performance in long context question answering and summarization. Furthermore, KV-Distill can be fine-tuned on domain-specific contexts to reduce context lengths by up 95% while preserving downstream task performance. We demonstrate the generalizability of KV-Distill across various model sizes and architectures.
BibTeX
@misc{chari2025kv,
author = {Vivek Chari and Guanghui Qin and Benjamin Van Durme},
title = {{KV-Distill: Nearly Lossless Context Compression for Transformers}},
year = {2025},
url = {https://openreview.net/forum?id=p7vJ3wsm34},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}