KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU

Di Chai, Leye Wang, Zeshen Su, Zhiguo Xia, Zhihang YuarXiv 2026 · arXiv

TL;DR

KVMEM is a virtual memory system for LLM agents that treats the key-value (KV) cache as a virtualizable resource, paging it across GPU memory, host RAM, and NVMe storage. By decoupling an agent's addressable workspace from physical GPU limits, KVMEM allows a consumer laptop GPU to seamlessly manage million-token histories while eliminating the massive re-computation overhead of traditional text-based retrieval.

What Problem Does This Paper Address?

As LLM agents tackle complex, long-horizon tasks—like software engineering or multi-turn project management—they accumulate massive interaction histories. This "workspace context" quickly hits two hard walls: the physical memory capacity of the GPU and the model's native context window limit.

To prevent out-of-memory errors, current systems rely on text compaction (summarizing old history) or text retrieval (storing history as text and pulling relevant snippets back into the prompt via RAG). Both approaches suffer from a coupled fidelity-efficiency problem:

  1. Compaction is lossy: Summaries often discard fine-grained details that only become critical many steps later.
  2. Retrieval is slow: Every time text is retrieved, the model must re-run the expensive "prefill" computation to generate the KV cache for tokens it has already processed.

KVMEM addresses this gap by asking: Why discard the KV cache we already computed? Instead of treating history as text, KVMEM treats it as virtualized memory pages.

Methods

KVMEM introduces a KV-context virtualization system implemented on top of a custom native C++/CUDA inference engine called QW3. It relies on three core mechanisms:

  • Step-Level Memory Scheduling: Instead of re-evaluating context relevance at every single token generation, KVMEM updates the active historical context exactly once per agent step (between the prefill and decode phases).
  • Query-Conditioned KV Retrieval: To find relevant history without scanning gigabytes of KV tensors, KVMEM creates a compact "Mean-K" index. It strips the positional encodings (RoPE) from the Key vectors of 32-token blocks and averages them. At retrieval time, the current query is matched against these lightweight block-level summaries directly in the model's attention space.
  • Tiered KV Management: KVMEM orchestrates a three-tier storage hierarchy (GPU $\leftrightarrow$ Host RAM $\leftrightarrow$ NVMe). It proactively stages out completed blocks asynchronously during chunked prefill, reuses overlapping pages across consecutive steps, and applies a delta re-RoPE kernel to dynamically adjust the positional encodings of recalled blocks so they fit seamlessly into their new positions in the active execution view.
Figure 1: Overview of KVMEM’s three core designs. First, step-level memory scheduling reselects a bounded working set once per agent step, after prefill and before decoding (Sec. 4.2). Second, query-conditioned retrieval represents the full history with compact block-level indexes, enabling efficient block selection with little GPU memory overhead (Sec. 4.3). Third, tiered KV management proactively stages out completed blocks, reuses overlapping GPU pages, retains frequently retrieved blocks in host memory, and packs the remaining GPU misses into a pipelined rematerialization path (Sec. 4.4).

Datasets

The authors evaluated KVMEM across several long-context agent benchmarks:

  • LongMemEval-S: A benchmark for multi-session chat assistants requiring evidence recovery from past interactions (~115K tokens/query).
  • MemoryAgentBench: Evaluates memory-dependent queries across long histories (up to 1.44M tokens).
  • AgentLongBench: Tests long simulated agent rollouts (ranging from 32K to 4M tokens).
  • DeepSWE v1.1: A repository-level software engineering benchmark used to test complete, unconstrained agent trajectories.

Key Insights and Ideas

  • Attention Shifts are Step-Bounded: By analyzing agent rollouts, the authors discovered that an LLM's attention distribution is highly stable within a single execution step but shifts sharply at step boundaries (averaging a 37.3× higher KL divergence across boundaries). This insight justifies updating the virtual memory layout only at step boundaries.
Attention stability and sparsity
  • Less is More (Mitigating "Lost in the Middle"): Surprisingly, on certain benchmarks, KVMEM achieved higher task success than using the model's full native context window. Feeding an LLM a massive context window can degrade its performance; by virtualizing the workspace and materializing only a highly relevant, bounded subset, KVMEM acts as an attention filter that actually helps the model reason better.

Key Results

  • Massive Speedups: KVMEM reduces post-compaction recovery latency by 11.4× to 53.8× compared to text-based RAG (Compact+RAG) because it completely bypasses historical text prefilling.
  • Local Consumer Deployment: On an off-the-shelf laptop equipped with a 24 GB RTX 5090 Laptop GPU, KVMEM successfully virtualized a 1-million-token workspace (4× the model's native 256K window) while sustaining an interactive generation speed of ~50 tokens/second.
  • Extreme Scalability: On server hardware, KVMEM scaled up to a 10-million-token workspace with a fixed 64K active execution view. The end-to-end Time-to-First-Token (TTFT) only increased from 0.43 seconds (at 256K tokens) to 1.60 seconds (at 10M tokens).
  • Downstream Agent Success: In full DeepSWE software engineering trials using Qwen3.8-27B, KVMEM boosted the average task success rate (Pass@1) from 43.8% to 48.4%, and dramatically improved task coverage (Pass@4) from 81.3% to 93.8% over the compaction baseline.

Limitations and Future Work

  • Active Window Boundedness: KVMEM virtualizes the addressable workspace, but it does not expand the number of tokens the model can jointly attend to in a single forward pass. Each step is still bounded by the active execution view budget.
  • Causal Context Mismatch: Reusing historical KV cache is not mathematically identical to full recomputation. Because the historical blocks were originally generated under a different causal context, there can be minor semantic drift, though evaluations show this does not hurt practical utility.
  • Storage Footprint: KV cache tensors are significantly larger than raw text, meaning KVMEM trades host RAM and NVMe capacity for execution speed.
  • API Constraints: Because KVMEM requires low-level control over KV allocation, page tables, and attention kernels, it cannot currently be used as a wrapper around black-box commercial LLM APIs (like OpenAI or Anthropic).

Summary updated

Abstract

Modern LLM agents operate in persistent workspaces whose accumulated history can exceed both GPU KV capacity and the model's native context window. Existing systems typically compact older context into summaries or retrieve it later as text, either losing fine-grained execution evidence or repeatedly prefilling content that the model has already processed. We present KVMem, a KV-context virtualization system that preserves overflowed workspace history as paged KV state across GPU memory, host memory, and NVMe. KVMem uses lightweight, model-native attention-space indexes to select relevant historical blocks and materializes a query-dependent execution view bounded by the model's native context window. Extensive evaluations on long-context agent benchmarks spanning histories up to one million tokens, including LongMemEval, MemoryAgentBench, and AgentLongBench, show that KVMem generally achieves higher task utility and greater inference efficiency than compaction-based approaches, the de facto standard for handling context overflow. In the DeepSWE long-context test with Qwen3.8-27B, KVMem improves task success from 43.8% with compaction-only context management to 48.4%. In our local-deployment evaluation, KVMem runs Qwen3.6/3.8-27B NVFP4 with MTP on an off-the-shelf laptop equipped with a 24\,GB RTX 5090 Laptop GPU, virtualizing agent workspaces of up to 1M tokens-four times the model's native 256K-token context window. In a single-session setting, KVMem generates $\sim$50 tokens/s, providing interactive responsiveness for local agent execution. More broadly, by decoupling addressable workspace size from the LLM's native context window, KVMem provides a practical path toward long-running agents whose workspaces can grow beyond that window.
BibTeX
@misc{chai2026kvmem,
  author = {Di Chai and Leye Wang and Zeshen Su and Zhiguo Xia and Zhihang Yu},
  title = {{KVMem: Virtualizing Million-Token Agent Workspaces on a Consumer GPU}},
  year = {2026},
  eprint = {2609.04852},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.04852},
}
Figure