Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction

Jang-Hyun Kim, Dongyoon Han, Sangdoo YunarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces Fast KVzip, a novel method for efficiently compressing the Key-Value (KV) cache in Large Language Models (LLMs) during inference. By employing lightweight "sink-attention" gates trained without backpropagation, Fast KVzip achieves high compression ratios (up to 70% reduction) with negligible performance degradation and computational overhead, making LLM deployment more practical.

What Problem Does This Paper Address?

Large Language Models (LLMs) rely heavily on the attention mechanism, which requires storing intermediate Key-Value (KV) states to avoid recomputation during inference. This KV cache grows linearly with the input sequence length, quickly becoming a memory bottleneck, especially for long-context tasks. Existing KV cache compression techniques often face a trade-off: either they significantly degrade model performance or they introduce substantial computational overhead during compression, hindering real-time applications. This paper aims to bridge this gap by developing a method that achieves both high compression and minimal performance loss with low computational cost.

Methods

The core innovation of Fast KVzip is a gating-based KV cache eviction mechanism. Instead of complex reconstruction or heuristic-based eviction, Fast KVzip uses lightweight sink-attention gating modules integrated into each attention layer. These gates, trained on frozen LLM weights, predict the importance of KV pairs directly from input hidden states. The training process is efficient, relying on a task-agnostic reconstruction objective derived from the original KVzip method, and avoids expensive backpropagation.

Key aspects of the method include:

  • Sink-Attention Gating: A novel architecture inspired by "attention sinks" that uses a fixed set of learnable "sink keys" to efficiently estimate KV importance.
  • Task-Agnostic Training: The gates are trained using a reconstruction-based objective, ensuring generalization across various downstream tasks without task-specific fine-tuning.
  • Efficient Training: The LLM weights are frozen, and only the gating parameters are trained, requiring minimal computational resources (less than one H100 GPU hour for 14B models).
  • Seamless Integration: The method integrates into both the prefill and decoding stages of LLM inference with minimal modifications. For decoding, a small buffer and parallel computation are used to mitigate latency overhead.

Datasets

  • Training Data: FineWeb-Edu dataset (Penedo et al., 2024) was used for training the gating mechanisms.
  • Evaluation Datasets:
    • Prefill-intensive: SCBench (Li et al., 2025) (including RULER, LongBench, and code comprehension tasks), OpenAI MRCR, SQUAD (Rajpurkar et al., 2016).
  • Decoding-intensive: AIME24, MATH (Hendrycks et al., 2021).
  • Benchmarks for comparison: KVPress (Devoto et al., 2025).

Key Insights and Ideas

  • Future KV utilization is predictable from hidden states: The paper's central insight is that the future importance of KV pairs can be directly inferred from the current hidden states, rather than requiring expensive recomputation of past context.
  • Attention sinks are effective for gating: The "sink-attention" architecture, inspired by attention sink mechanisms, proves to be a robust and efficient way to design the gating modules.
  • Task-agnostic training is crucial for generalization: Training gates using a reconstruction objective on a general corpus (FineWeb-Edu) leads to strong performance across diverse tasks, avoiding the pitfalls of task-specific overfitting seen in some prior methods.
  • Hidden states are superior gate inputs: Empirical analysis shows that using hidden states as input to the gating mechanism yields better performance than using key states or pre-RoPE key states.
  • Recency bias is important: Retaining KV features from recent tokens improves performance, guiding the eviction strategy.

Key Results

  • High Compression Ratio: Fast KVzip achieves near-lossless performance while evicting up to 70% of the KV cache.
  • Negligible Overhead: The method incurs negligible computational cost and memory overhead during inference.
  • Performance Preservation: Maintains near-lossless performance across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning.
  • Efficiency Gains: Significantly reduces peak memory usage and prefill time compared to KVzip and no-compression baselines. For instance, it matches KVzip's accuracy at a 30-40% KV cache budget ratio, while KVzip requires twice the prefill computation.
  • Generality: Demonstrates effectiveness across diverse LLM families (Qwen2.5, Qwen3, Gemma3) and architectures, including those with quantized weights and hybrid attention mechanisms.
  • Faster Training: Gate training is highly efficient, requiring less than one H100 GPU hour for 14B models.

Limitations and Future Work

  • Limitations:
    • The paper notes that while their method is task-agnostic, some prior methods are specifically designed for reasoning tasks, making direct comparisons challenging in those specific niches.
  • The current gating mechanism is primarily focused on eviction.
  • Future Work:
    • Structured Computation Patterns: Exploring how to integrate gating during pretraining to induce hardware-efficient structured computation patterns.
  • Unified Framework: Developing a unified framework that integrates various gating approaches.
  • Multi-Choice Decisions: Extending the gating mechanism to support more complex decisions, such as skipping computation, discarding KV features, or storing them for selective retrieval.

Summary updated

Abstract

Efficient key-value (KV) cache management is crucial for the practical deployment of large language models (LLMs), yet existing compression techniques often incur a trade-off between performance degradation and computational overhead. We propose a novel gating-based KV cache eviction method for frozen-weight LLMs that achieves high compression ratios with negligible computational cost. Our approach introduces lightweight sink-attention gating modules to identify and retain critical KV pairs, and integrates seamlessly into both the prefill and decoding stages. The proposed gate training algorithm relies on forward passes of an LLM, avoiding expensive backpropagation, while achieving strong task generalization through a task-agnostic reconstruction objective. Extensive experiments across the Qwen2.5-1M, Qwen3, and Gemma3 families show that our method maintains near-lossless performance while evicting up to 70% of the KV cache. The results are consistent across a wide range of tasks, including long-context understanding, code comprehension, and mathematical reasoning, demonstrating the generality of our approach.
BibTeX
@misc{kim2026fast,
  author = {Jang-Hyun Kim and Dongyoon Han and Sangdoo Yun},
  title = {{Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction}},
  year = {2026},
  eprint = {2601.17668},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.17668},
}
Figure