TL;DR
TripleKV is a three-tier KV cache management system designed to handle the massive memory demands of AI agents. By using tool-aware latency prediction to prefetch context from SSDs into host DRAM—and protecting that data from premature eviction—it reduces tail latency (P95 TTFT) by up to 49% compared to standard tiered caching systems like vLLM.
What Problem Does This Paper Address?
Modern AI agents typically follow the ReAct (Reasoning and Action) pattern: they perform LLM inference, execute a tool (like a web search or code execution), and then return to inference with an expanded context.
This creates two major bottlenecks:
- Memory Explosion: Contexts grow to tens of thousands of tokens. For a 30B parameter model, a single 50K-token context can occupy 4.6 GiB. A single H100 GPU can only hold a handful of these in its high-bandwidth memory (HBM).
- The "Cold Start" Problem: When multiple agents run concurrently, the system must evict idle KV caches to DRAM or SSD to make room. When an agent finishes a tool call and resumes inference, retrieving its context from an SSD can take seconds, causing significant stalls.
Existing solutions either "pin" context in GPU memory (wasting space) or prefetch directly into the GPU (causing contention with active requests).
Methods
TripleKV introduces a "Predict, Prefetch, Protect" framework built on top of the popular vLLM engine. It treats GPU HBM, host DRAM, and SSD as a three-tier hierarchy.
- Predict: The system uses a tool-aware latency predictor. Unlike previous methods that use simple moving averages, TripleKV looks at tool types, input arguments, and even "live cues" (like partial output from a long-running tool) to estimate exactly when a tool call will finish.
- Prefetch: Once the end-time is estimated, a prefetch engine asynchronously moves the agent's KV blocks from the SSD into host DRAM. Crucially, it avoids prefetching directly into the GPU to prevent interfering with other active inference tasks.
- Protect: Standard Least Recently Used (LRU) policies often evict prefetched data before the agent actually resumes. TripleKV implements a multi-queue replacement policy in DRAM that prioritizes keeping prefetched and "backup" blocks (copies of what's currently in the GPU) safe from eviction. It also acts as a "victim cache," refilling DRAM from the SSD the moment a block is kicked out of the GPU.
Datasets
The authors evaluated TripleKV using four diverse agentic benchmarks, replaying traces of 50 random tasks each to ensure consistency:
- DeepResearch Bench (DRB): Open-ended research questions requiring web visits and summaries.
- GAIA: Multi-step web skills and tool use.
- Terminal-Bench 4.0 (TB4): Software engineering tasks performed in a terminal environment.
- BrowseComp-Plus (BC+): Complex multi-hop web search queries.
Key Insights and Ideas
- The "What" is Known, the "When" is Not: In agentic workloads, the system knows exactly which KV blocks will be needed next (the existing prefix). The only uncertainty is the timing of the tool call completion.
- DRAM is the Ideal Buffer: Prefetching to the GPU is risky because GPU memory is the scarcest resource. Prefetching to DRAM hides the massive SSD-to-DRAM latency (seconds) while leaving the final DRAM-to-GPU transfer (milliseconds) to the moment of request admission.
- LRU is Counter-Productive: In high-concurrency scenarios, the "residency time" of a block in cache is often shorter than the tool execution time. This means standard LRU is almost guaranteed to evict the data you are about to need.
Key Results
TripleKV significantly outperforms both standard vLLM and recent agent-aware baselines like TokenCake and Continuum:
- Reduced SSD Stalls: The fraction of requests requiring slow, critical-path SSD reads dropped from a baseline of 45–76% to under 10%.
- Tail Latency: It achieved up to a 49% reduction in P95 Time to First Token (TTFT) compared to vLLM with tiered storage. Compared to other specialized agent systems, it was 1.5–7× faster at the tail.
- Throughput: By reducing the time requests spend hogging GPU slots while waiting for data, TripleKV improved token throughput by up to 40%.
Limitations and Future Work
- DRAM Capacity Limits: While TripleKV manages DRAM efficiently, performance still degrades if the total working set of all active agents exceeds the host's DRAM capacity (as seen in the GAIA and BC+ benchmarks at high concurrency).
- Predictor Sensitivity: While the tool-aware predictor is highly accurate (21% Mean Absolute Percentage Error), very unpredictable tools could still lead to "late" prefetches that land on the critical path.
- Future Directions: The authors suggest exploring disaggregated memory systems or even faster storage interconnects to further scale the number of concurrent agents a single GPU can support.
Summary updated