Introduction
As Large Language Models (LLMs) become central to interactive AI, serving costs and latency have become critical bottlenecks. In multi-turn conversations, models must maintain context by referencing previous dialogue. Current serving systems typically manage this by storing intermediate mathematical states known as Key-Value (KV) caches. However, due to the high cost of GPU memory (HBM), these caches are often discarded between turns, forcing the system to reprocess the entire conversation history for every new message. This paper introduces CachedAttention (and its underlying storage system, AttentionStore), a mechanism designed to eliminate this redundant computation.
The Problem: Redundant Recomputation
The primary inefficiency in modern LLM serving is the "prefilling" phase. When a user sends a new prompt in an ongoing session, the GPU must re-calculate the KV cache for all previous tokens before it can generate the first token of the new response.
Analysis of real-world datasets like ShareGPT reveals that as conversations progress, the vast majority of GPU cycles are spent on these historical tokens. In long sessions, up to 99% of the prefilling cost is purely redundant. This waste scales linearly with conversation length, leading to significant delays for users and inflated operational costs for providers.
The Solution: CachedAttention
CachedAttention transforms KV caches from transient data into persistent assets. Instead of discarding the cache after a response is generated, the system offloads it to a hierarchical storage layer. When the user returns with a follow-up question, the system retrieves the existing cache rather than recomputing it.
Key Technical Innovations
To make this approach viable, the authors implemented three core strategies:
- AttentionStore (Hierarchical Caching): A tiered storage system moves caches from fast GPU memory to Host RAM and eventually to SSDs. This allows the system to support thousands of concurrent, inactive sessions at a fraction of the cost of HBM.
- Latency Hiding (Layer-wise Pre-loading): To mitigate the slower transfer speeds of SSDs and RAM, CachedAttention uses a pipelined approach. While the GPU computes one layer of the model, the system concurrently fetches the cache for the subsequent layer. This overlaps data transfer with computation, effectively "hiding" the storage bottleneck from the user.
- Decoupled Positional Encoding: Standard LLMs bake token positions (e.g., RoPE) directly into the KV cache. This makes caches fragile; if the beginning of a conversation is truncated, the entire cache becomes invalid. CachedAttention saves "raw" KV pairs and applies positional math only during the loading phase, ensuring the cache remains flexible and reusable even as context windows shift.
Evaluation and Results
The researchers evaluated CachedAttention using the ShareGPT dataset across various models, including LLaMA-2 (70B) and Mistral-7B. The results demonstrate a transformative impact on performance:
- Latency: Time to First Token (TTFT) was reduced by up to 87%, providing near-instant responses even in deep conversation histories.
- Throughput: Prefilling throughput improved by up to 7.8×, allowing hardware to support significantly more users.
- Cost Efficiency: By shifting the storage burden to cheaper RAM and SSDs, the total inference cost was slashed by up to 70%.
Conclusion
CachedAttention addresses a fundamental inefficiency in LLM serving by treating conversation history as a reusable resource. By combining hierarchical storage with clever latency-hiding techniques and decoupled positional encoding, it provides a scalable path toward more responsive and affordable AI interactions. Future work will likely focus on cache compression and further optimizing disk bandwidth for extreme-scale deployments.
Summary updated