TripleKV: Predict, Prefetch, Protect -- KV Cache Management for AI Agents

Shashwat Jaiswal, Chloe Alverti, Mengmei Ye, Chen Wang, Hubertus Franke, Fan Lai, Tianyin Xu, Josep TorrellasarXiv 2026 · arXiv

TL;DR

TripleKV is a three-tier KV cache management system designed to handle the massive memory demands of AI agents. By using tool-aware latency prediction to prefetch context from SSDs into host DRAM—and protecting that data from premature eviction—it reduces tail latency (P95 TTFT) by up to 49% compared to standard tiered caching systems like vLLM.

What Problem Does This Paper Address?

Modern AI agents typically follow the ReAct (Reasoning and Action) pattern: they perform LLM inference, execute a tool (like a web search or code execution), and then return to inference with an expanded context.

Figure 1: The ReAct agent loop.

This creates two major bottlenecks:

  1. Memory Explosion: Contexts grow to tens of thousands of tokens. For a 30B parameter model, a single 50K-token context can occupy 4.6 GiB. A single H100 GPU can only hold a handful of these in its high-bandwidth memory (HBM).
  2. The "Cold Start" Problem: When multiple agents run concurrently, the system must evict idle KV caches to DRAM or SSD to make room. When an agent finishes a tool call and resumes inference, retrieving its context from an SSD can take seconds, causing significant stalls.

Existing solutions either "pin" context in GPU memory (wasting space) or prefetch directly into the GPU (causing contention with active requests).

Methods

TripleKV introduces a "Predict, Prefetch, Protect" framework built on top of the popular vLLM engine. It treats GPU HBM, host DRAM, and SSD as a three-tier hierarchy.

  • Predict: The system uses a tool-aware latency predictor. Unlike previous methods that use simple moving averages, TripleKV looks at tool types, input arguments, and even "live cues" (like partial output from a long-running tool) to estimate exactly when a tool call will finish.
  • Prefetch: Once the end-time is estimated, a prefetch engine asynchronously moves the agent's KV blocks from the SSD into host DRAM. Crucially, it avoids prefetching directly into the GPU to prevent interfering with other active inference tasks.
  • Protect: Standard Least Recently Used (LRU) policies often evict prefetched data before the agent actually resumes. TripleKV implements a multi-queue replacement policy in DRAM that prioritizes keeping prefetched and "backup" blocks (copies of what's currently in the GPU) safe from eviction. It also acts as a "victim cache," refilling DRAM from the SSD the moment a block is kicked out of the GPU.
Figure 3: Timeline of one agent turn in various systems.

Datasets

The authors evaluated TripleKV using four diverse agentic benchmarks, replaying traces of 50 random tasks each to ensure consistency:

  • DeepResearch Bench (DRB): Open-ended research questions requiring web visits and summaries.
  • GAIA: Multi-step web skills and tool use.
  • Terminal-Bench 4.0 (TB4): Software engineering tasks performed in a terminal environment.
  • BrowseComp-Plus (BC+): Complex multi-hop web search queries.

Key Insights and Ideas

  • The "What" is Known, the "When" is Not: In agentic workloads, the system knows exactly which KV blocks will be needed next (the existing prefix). The only uncertainty is the timing of the tool call completion.
  • DRAM is the Ideal Buffer: Prefetching to the GPU is risky because GPU memory is the scarcest resource. Prefetching to DRAM hides the massive SSD-to-DRAM latency (seconds) while leaving the final DRAM-to-GPU transfer (milliseconds) to the moment of request admission.
  • LRU is Counter-Productive: In high-concurrency scenarios, the "residency time" of a block in cache is often shorter than the tool execution time. This means standard LRU is almost guaranteed to evict the data you are about to need.

Key Results

TripleKV significantly outperforms both standard vLLM and recent agent-aware baselines like TokenCake and Continuum:

  • Reduced SSD Stalls: The fraction of requests requiring slow, critical-path SSD reads dropped from a baseline of 45–76% to under 10%.
  • Tail Latency: It achieved up to a 49% reduction in P95 Time to First Token (TTFT) compared to vLLM with tiered storage. Compared to other specialized agent systems, it was 1.5–7× faster at the tail.
  • Throughput: By reducing the time requests spend hogging GPU slots while waiting for data, TripleKV improved token throughput by up to 40%.
TripleKV significantly reduces the number of cache misses caused by capacity and replacement issues.

Limitations and Future Work

  • DRAM Capacity Limits: While TripleKV manages DRAM efficiently, performance still degrades if the total working set of all active agents exceeds the host's DRAM capacity (as seen in the GAIA and BC+ benchmarks at high concurrency).
  • Predictor Sensitivity: While the tool-aware predictor is highly accurate (21% Mean Absolute Percentage Error), very unpredictable tools could still lead to "late" prefetches that land on the critical path.
  • Future Directions: The authors suggest exploring disaggregated memory systems or even faster storage interconnects to further scale the number of concurrent agents a single GPU can support.

Summary updated

Abstract

LLM agents alternate between inference and tool execution, repeatedly reusing their growing context across calls. Under high concurrency, their combined key-value (KV) cache working set can reach hundreds of GiB, far exceeding GPU memory capacity. As idle KV cache blocks are evicted to slower storage tiers, returning agents incur expensive retrievals that stall inference. To address this problem, we present \emph{TripleKV}, a three-tier KV cache management system that hides the delays of these retrievals. %exploits tool execution to hide these delays. In agentic workloads, the KV blocks needed by the next inference request are known in advance; only the time of their reuse is uncertain. TripleKV uses a tool-aware latency predictor to estimate when each tool call will complete and prefetches the required KV blocks from SSD into host DRAM ahead of time. Unlike GPU prefetching, this avoids reserving scarce GPU memory and interfering with concurrent inference requests. To further improve DRAM utilization, TripleKV eliminates duplication of GPU-resident blocks in DRAM, and introduces a victim-cache mechanism that refills blocks from SSD upon their eviction from GPU memory. Across data, web, and coding agents, TripleKV reduces the fraction of requests requiring critical-path SSD reads from 45--76\% to under 10\%. It also reduces P95 time to first token (TTFT) by up to 49\% compared with vLLM extended with DRAM and SSD KV-cache tiers, and achieves 1.5-7$\times$ lower P95 TTFT than recent agent-aware systems.
BibTeX
@misc{jaiswal2026triplekv,
  author = {Shashwat Jaiswal and Chloe Alverti and Mengmei Ye and Chen Wang and Hubertus Franke and Fan Lai and Tianyin Xu and Josep Torrellas},
  title = {{TripleKV: Predict, Prefetch, Protect -- KV Cache Management for AI Agents}},
  year = {2026},
  eprint = {2611.25112},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25112},
}
Figure