EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies

Yuhao Liu, Yuehao Li, Linghe Zhang, Gaobin Huang, Junran Li, Hongchang Wang, Haifang JianarXiv 2026 · DOI · arXiv

TL;DR

EarlySelectKV is a training-free method that accelerates long-context LLM inference by precomputing Key-Value (KV) selection indices. By using "proxy queries" derived from earlier hidden states, it moves the time-consuming selection process off the critical path, achieving up to a 12.4% throughput boost and 11.1% GPU memory savings with negligible impact on model accuracy.

What Problem Does This Paper Address?

As Large Language Models (LLMs) handle increasingly massive contexts, the "KV cache"—the memory used to store previous tokens' keys and values—becomes a massive bottleneck. Dynamic sparse attention methods attempt to solve this by only attending to a small, relevant subset of the KV cache for each new token.

However, these methods face a "serial dependency" problem. To know which KV entries are important, the model must first compute the query for the current layer. Only then can it score candidates, select indices, and finally perform attention. As we make attention more sparse (and thus faster), this selection overhead (or "routing") starts to dominate the total latency.

Figure 1: Conventional query-conditioned KV selection and EARLYSELECTKV. Conventional selectors wait for the target-layer query, so index generation remains on the immediate pre-attention path. EarlySelectKV derives a proxy query from preceding hidden states and precomputes KV indices in an earlier computation window.

Methods

The authors introduce EarlySelectKV, a framework that transforms serial KV selection into an anticipatory, parallel process.

  1. Proxy Queries: Instead of waiting for the "real" query at layer $i+1$, the system uses the hidden states from layer $i$ to generate a proxy query.
  2. Side Flow: This proxy query is used in a "side flow" to precompute which KV pages or tokens will be needed for the next layer while the current layer is still busy with its own computations.
  3. Anchor Selection: The authors explore different "anchors" for these proxies:
  4. ES-In: Uses the input to the current block (maximum preparation time).
  5. ES-Mid: Uses the state after the current layer's attention but before the Feed-Forward Network (higher fidelity).
  6. ES-Mix: A model-specific blend that chooses the best anchor for each layer based on offline sensitivity profiling.
  7. Training-Free: Crucially, this requires no retraining. It can be "plugged in" to existing sparse attention selectors like RocketKV, Quest, or Loki.
The architecture of the Main Flow and Side Flow.

Datasets

The researchers evaluated EarlySelectKV across a diverse battery of long-context benchmarks:

  • LongBench: For general long-document understanding.
  • Needle-in-a-Haystack (NIAH) & Passkey: To test long-range information retrieval.
  • RULER: A suite for complex synthetic reasoning (retrieval, tracking, counting).
  • MATH-500: To ensure the method doesn't break mathematical reasoning during independent generation.

Key Insights and Ideas

  • Fidelity vs. Timing: The "aha moment" is realizing that routing and attention require different levels of query precision. While attention needs the exact query to calculate precise weights, routing only needs to preserve the relative preference among candidates. Proxies from earlier layers are "good enough" for selection.
  • Routing as an Optimization Target: Traditionally, researchers focused on what to select. This paper argues that when you select is just as important for system-level performance.
  • CPU Offloading: Because indices are precomputed, the routing logic and metadata can be offloaded to the CPU. This frees up significant GPU memory without stalling the GPU's attention computation.
Figure 2: Theoretical routing overhead under sparse attention budgets. We analyze a single Llama-3.1-8B layer with an 8K KV cache and vary the selected-token budget. As the budget decreases, sparse attention be- comes cheaper, and routing accounts for a larger fraction of the remaining attention-side computation.

Key Results

  • Performance Parity: Across LongBench, ES-RocketKV matched the original RocketKV within +0.018 points. On RULER, the difference was a negligible -0.01 points.
  • Throughput Gains: In a real-world deployment using TensorRT-LLM, EarlySelectKV improved end-to-end throughput by up to 12.4% for long sequences (8192 output tokens).
  • Memory Efficiency: By offloading routing states to the CPU, the method reduced GPU-resident memory by 11.1% of the total KV/summary pool (approx. 5.3 GiB in their configuration).
  • Broad Applicability: The method successfully improved multiple different selectors (Quest, Loki, RocketKV) across various model families (Llama 3/3.1, Mistral, Qwen 2.5).

Limitations and Future Work

  • Preparation Window: The benefits are most pronounced when the "preparation window" (the time between the proxy and the real query) is large enough to hide the routing latency. For very short outputs or extremely expensive routers, the gains might diminish.
  • Batching Effects: While throughput gains persist under batching for long decodes, the benefits approach zero for very short requests in large batches due to fixed synchronization costs.
  • Static Profiling: Currently, the layer-wise anchor assignment (ES-Mix) is done offline. Future work could explore dynamic, online adjustment of these anchors.
  • Co-adaptation: The authors suggest that incorporating EarlySelectKV during pre-training or fine-tuning could further improve proxy fidelity by allowing the model to learn to produce better "anticipatory" hidden states.

Summary updated

Abstract

Query-conditioned sparse KV selection reduces long-context inference cost by attending to a subset of historical KV entries at each decoding step. Existing selectors must wait for the target-layer query before scoring candidates and generating indices, placing routing on the immediate pre-attention path. We introduce EarlySelectKV, a training-free transformation that uses proxy queries from earlier hidden states to precompute selected KV indices while retaining the real target-layer query for final attention. Tests with RocketKV, Quest, and Loki show no consistent pattern of quality degradation across the evaluated settings. In the primary RocketKV evaluation, ES-RocketKV differs by +0.018 points on LongBench, approximately -0.01 points on RULER, and +0.25 percentage points on representative NIAH/Passkey settings. In a TensorRT-LLM deployment with CPU-offloaded routing, ES-RocketKV improves end-to-end output throughput by up to 12.4% and reduces GPU-resident memory by 11.1% of the combined KV/K-summary pool. This makes routing timing an explicit optimization target alongside KV subset selection in sparse long-context decoding. Code is available at https://github.com/sea-with-sakura/EarlySelectKV.
BibTeX
@misc{liu2026earlyselectkv,
  author = {Yuhao Liu and Yuehao Li and Linghe Zhang and Gaobin Huang and Junran Li and Hongchang Wang and Haifang Jian},
  title = {{EarlySelectKV: Anticipatory KV Selection via Cross-Layer Query Proxies}},
  year = {2026},
  eprint = {2611.01715},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01715},
}
Figure