LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation

Jinwoo Ahn, Ingyu Seong, Akhil Kedia, Junhan Kim, Hyemi Jang, Kangwook Lee, Yongkweon Jeon

Abstract

Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token.

Introduction

As LLMs are increasingly deployed for long-context tasks—such as analyzing entire codebases or long legal documents—the memory requirements of the Key-Value (KV) cache have become a critical bottleneck. The KV cache stores intermediate computations to speed up generation, but its size scales linearly with sequence length. For instance, a 1-million-token context on a Llama-3-70B model would require approximately 320GB of memory, exceeding the capacity of most modern hardware.

To manage this, researchers employ KV cache eviction, which involves removing "unimportant" tokens from the cache. Current methods generally fall into two categories:

  1. Simple Heuristics (e.g., SnapKV): These methods are computationally efficient but "blind," as they estimate token importance based solely on the input prompt without considering the model's future output.
  2. Draft-based Methods (e.g., SpecKV, LAQ): These methods "glimpse the future" by using a smaller draft model to generate a surrogate response. While more accurate, the overhead of generating this draft significantly increases the Time-to-First-Token (TTFT), making them impractical for latency-sensitive applications.

LookaheadKV bridges this gap by providing the accuracy of future-aware methods with the speed of simple heuristics.

Methodology: Glimpsing the Future without Generation

LookaheadKV replaces explicit text generation with a lightweight prediction module that simulates the attention patterns of a future response.

Overview of LookaheadKV training and inference architecture, illustrating the use of learnable lookahead tokens and LoRA adapters to predict importance scores.

Learnable Lookahead Tokens and LoRA

Instead of generating actual text tokens, LookaheadKV appends a set of learnable "soft" tokens to the input sequence. These tokens are mathematical representations trained to act as a proxy for the model's future response. To process these tokens without interfering with the model's standard processing of the user prompt, the framework utilizes Lookahead LoRA (Low-Rank Adaptation). These adapters are selectively activated only for the lookahead tokens during the prefill stage.

Importance Distillation

During training, the system calculates the "true" importance scores by observing the attention weights of the model's actual generated response. LookaheadKV then uses a KL-divergence loss to train the soft tokens and LoRA modules to predict these true scores. This allows the model to learn the salience of input tokens—where it is likely to look in the future—without the computational burden of autoregressive generation.

Experimental Setup

The framework was evaluated across several model families, including Llama-3 and Qwen, using a diverse training mixture of 86K samples covering instruction following, code, and mathematical reasoning. Performance was measured on comprehensive long-context benchmarks:

  • LongBench: Multi-task long-context understanding.
  • RULER: Needle-in-a-haystack retrieval tasks.
  • MT-Bench: Multi-turn conversation quality.

Key Results

LookaheadKV achieves a superior balance between accuracy and latency compared to existing state-of-the-art methods.

Comparison of QASPER score vs. Time-to-First-Token (TTFT), showing LookaheadKV achieving high accuracy with latency nearly identical to the theoretical minimum.

Performance and Efficiency

  • Latency Reduction: LookaheadKV reduces eviction overhead by up to 14.5x compared to draft-based methods like LAQ. Its actual runtime overhead is less than 2.16% of the total prefill time, making it nearly as fast as simple heuristics like SnapKV.
  • High Accuracy: On the LongBench and RULER benchmarks, LookaheadKV consistently outperformed both simple heuristics and draft-based methods. It is particularly effective in "low-budget" scenarios where only a small fraction of the KV cache can be retained.
  • Generalization: Although trained on sequences up to 16K tokens, the modules generalized effectively to contexts as large as 128K tokens, indicating that the learned importance patterns are scale-invariant.

Conclusion

LookaheadKV demonstrates that explicit generation is not required to accurately predict KV cache importance. By leveraging learnable soft tokens and parameter-efficient adapters, LookaheadKV provides a practical solution for long-context LLM inference, offering significant memory savings and high accuracy with negligible impact on latency. Future work includes extending this approach to decoding-stage eviction and scaling the training to ultra-large model architectures.

Summary updated

Abstract

Transformer-based large language models (LLMs) rely on key-value (KV) caching to avoid redundant computation during autoregressive inference. While this mechanism greatly improves efficiency, the cache size grows linearly with the input sequence length, quickly becoming a bottleneck for long-context tasks. Existing solutions mitigate this problem by evicting prompt KV that are deemed unimportant, guided by estimated importance scores. Notably, a recent line of work proposes to improve eviction quality by "glimpsing into the future", in which a draft generator produces a surrogate future response approximating the target model's true response, and this surrogate is subsequently used to estimate the importance of cached KV more accurately. However, these approaches rely on computationally expensive draft generation, which introduces substantial prefilling overhead and limits their practicality in real-world deployment. To address this challenge, we propose LookaheadKV, a lightweight eviction framework that leverages the strength of surrogate future response without requiring explicit draft generation. LookaheadKV augments transformer layers with parameter-efficient modules trained to predict true importance scores with high accuracy. Our design ensures negligible runtime overhead comparable to existing inexpensive heuristics, while achieving accuracy superior to more costly approximation methods. Extensive experiments on long-context understanding benchmarks, across a wide range of models, demonstrate that our method not only outperforms recent competitive baselines in various long-context understanding tasks, but also reduces the eviction cost by up to 14.5x, leading to significantly faster time-to-first-token. Our code is available at https://github.com/SamsungLabs/LookaheadKV.
BibTeX
@inproceedings{ahn2026lookaheadkv,
  author = {Jinwoo Ahn and Ingyu Seong and Akhil Kedia and Junhan Kim and Hyemi Jang and Kangwook Lee and Yongkweon Jeon},
  title = {{LookaheadKV: Fast and Accurate KV Cache Eviction by Glimpsing into the Future without Generation}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=RVLMGPXt2i},
}
Figure