TL;DR
TideCache is a novel serving system designed for hybrid LLMs (like Kimi K3 and Qwen 3.8) running on CPU-GPU superchips. By dynamically repartitioning GPU memory and streaming KV caches layer-wise from host memory during "compute gaps," it achieves up to 2.04× higher throughput and reduces latency by up to 59% compared to state-of-the-art baselines.
What Problem Does This Paper Address?
Modern "hybrid" Large Language Models (LLMs) are becoming the standard for open-weight releases. Unlike traditional Transformers that use global attention for every layer, hybrid models mix global-attention layers (where memory usage grows with context length) with bounded-state layers like Mamba or sliding-window attention (where memory usage is fixed).
Serving these models efficiently is a nightmare for two reasons:
- Heterogeneous Cache Demands: Different tasks stress different parts of the memory. A long-document analysis needs massive global-KV cache, while a high-concurrency chat service needs many small bounded-state slots. Static memory partitioning—reserving a fixed slice for each—inevitably leads to one pool being empty while the other sits idle.
- The Memory Wall: Even with massive HBM (High Bandwidth Memory), long-context agentic workflows quickly exhaust GPU capacity, forcing expensive recomputation or slow offloading.
Methods
TideCache introduces three core components to turn CPU-GPU superchips (like NVIDIA’s Grace Hopper) into a unified, flexible memory hierarchy:
- Transparent Cache Pool (TCP): Instead of static partitions, TCP manages a shared arena of physical GPU memory pages. It uses the CUDA virtual memory API to map and unmap physical pages behind fixed virtual addresses. This allows the system to "steal" memory from an underutilized bounded-state pool and give it to a struggling global-KV pool without changing how the underlying kernels or CUDA graphs access data.
- Cache Streamer: This is the "aha" moment of the paper. Because bounded-state layers don't need the global KV cache to compute, they create a "computation interval." TideCache uses this time to prefetch the next global-attention layer's data from the CPU host memory into a small, shared "streaming slot" on the GPU. By double-buffering these transfers, the system hides the latency of moving data from the CPU.
- Request Scheduler: A high-level controller that monitors "memory pressure." It decides whether a new request should run entirely on the GPU ("resident") or utilize the streaming pipeline. It also triggers the TCP to rebalance memory between epochs.
Datasets
The authors evaluated TideCache using synthetic workloads derived from two major sources:
- ShareGPT: Used to simulate general chat traffic with varying input/output lengths.
- Claude Code Traces: Used to simulate "coding agent" behavior, characterized by very long contexts (40K–110K tokens) and multiple interaction turns where context is reused.
Evaluation was performed across seven model/platform configurations, including models like Kimi K3, Ring-2.6-1T, and Qwen3.8-27B on NVIDIA GH200 and GB300 superchips.
Key Insights and Ideas
- Exploiting the "Compute Gap": The most elegant insight is that the very thing that makes hybrid models efficient (bounded-state layers) also provides the perfect temporal window to move data. While the GPU is busy with a Mamba layer, the NVLink-C2C interconnect is free to fetch the next Attention layer's prefix.
- Transparency is King: Previous dynamic sharing schemes often required rewriting attention kernels or complex pointer management. By operating at the virtual memory page-table level, TideCache remains "transparent" to the rest of the software stack (like SGLang), making it much easier to integrate into production systems.
- Streaming as Capacity Expansion: Streaming isn't just a fallback for when you run out of memory; it effectively expands the "virtual" size of the GPU's HBM. The authors show that streaming can increase effective global-KV capacity by ~77% for large models.
Key Results
- Throughput: TideCache completes up to 2.04× as many agent turns compared to SGLang with HiCache (a state-of-the-art offloading baseline).
- Latency: It reduces the median (P50) per-token latency by 20–59% and improves the P50 Time-To-First-Token (TTFT) for agents by 1.40–3.05×.
- Adaptability: In shifting workloads (moving from chat bursts to agent bursts), TideCache's dynamic repartitioning allowed it to complete 59% more turns than static baselines, which hit a "memory wall" and were forced to queue requests.
Limitations and Future Work
- Hardware Specificity: TideCache is heavily optimized for "superchips" with high-bandwidth CPU-GPU interconnects (like NVLink-C2C). Its benefits would likely be significantly diminished on standard PCIe-based systems where the "compute gap" isn't wide enough to hide the transfer latency.
- Prefill vs. Decode: While the paper focuses heavily on the decode phase, the authors note that prefill computation still occupies the GPU and can delay subsequent requests. Future work could look at better interleaving of prefill and streamed decode tasks.
- Kernel Integration: While the system is "transparent," the authors suggest that even further gains could be found by co-designing attention kernels that are aware of the streaming slots to reduce memory fragmentation.
Summary updated