TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming

Jinbin Luo, Xiaozhe Yao, Shang Yang, Zhijian Liu, Song Han, Qinghao HuarXiv 2026 · arXiv

TL;DR

TideCache is a novel serving system designed for hybrid LLMs (like Kimi K3 and Qwen 3.8) running on CPU-GPU superchips. By dynamically repartitioning GPU memory and streaming KV caches layer-wise from host memory during "compute gaps," it achieves up to 2.04× higher throughput and reduces latency by up to 59% compared to state-of-the-art baselines.

What Problem Does This Paper Address?

Modern "hybrid" Large Language Models (LLMs) are becoming the standard for open-weight releases. Unlike traditional Transformers that use global attention for every layer, hybrid models mix global-attention layers (where memory usage grows with context length) with bounded-state layers like Mamba or sliding-window attention (where memory usage is fixed).

Serving these models efficiently is a nightmare for two reasons:

  1. Heterogeneous Cache Demands: Different tasks stress different parts of the memory. A long-document analysis needs massive global-KV cache, while a high-concurrency chat service needs many small bounded-state slots. Static memory partitioning—reserving a fixed slice for each—inevitably leads to one pool being empty while the other sits idle.
  2. The Memory Wall: Even with massive HBM (High Bandwidth Memory), long-context agentic workflows quickly exhaust GPU capacity, forcing expensive recomputation or slow offloading.
Figure 2: Illustration of a typical hybrid LLM architecture. Global- attention layers (full, DSA or QSA (Qiu et al., 2026)) hold a global- KV that grows with the context; bounded-state layers (sliding window, GDN, KDA) hold a window KV or recurrent state of fixed size per request. All cache components share GPU memory. sharing an arena across cache layouts can require changes to indexing and memory access, complicating integration with attention backends and other serving features.

Methods

TideCache introduces three core components to turn CPU-GPU superchips (like NVIDIA’s Grace Hopper) into a unified, flexible memory hierarchy:

  • Transparent Cache Pool (TCP): Instead of static partitions, TCP manages a shared arena of physical GPU memory pages. It uses the CUDA virtual memory API to map and unmap physical pages behind fixed virtual addresses. This allows the system to "steal" memory from an underutilized bounded-state pool and give it to a struggling global-KV pool without changing how the underlying kernels or CUDA graphs access data.
  • Cache Streamer: This is the "aha" moment of the paper. Because bounded-state layers don't need the global KV cache to compute, they create a "computation interval." TideCache uses this time to prefetch the next global-attention layer's data from the CPU host memory into a small, shared "streaming slot" on the GPU. By double-buffering these transfers, the system hides the latency of moving data from the CPU.
  • Request Scheduler: A high-level controller that monitors "memory pressure." It decides whether a new request should run entirely on the GPU ("resident") or utilize the streaming pipeline. It also triggers the TCP to rebalance memory between epochs.
Figure 4: TideCache overview: the Request Scheduler controls resident- and streaming-region admission; the Transparent Cache Pool manages cache backing, and the Cache Streamer feeds shared GPU slots from host memory.

Datasets

The authors evaluated TideCache using synthetic workloads derived from two major sources:

  • ShareGPT: Used to simulate general chat traffic with varying input/output lengths.
  • Claude Code Traces: Used to simulate "coding agent" behavior, characterized by very long contexts (40K–110K tokens) and multiple interaction turns where context is reused.

Evaluation was performed across seven model/platform configurations, including models like Kimi K3, Ring-2.6-1T, and Qwen3.8-27B on NVIDIA GH200 and GB300 superchips.

Key Insights and Ideas

  • Exploiting the "Compute Gap": The most elegant insight is that the very thing that makes hybrid models efficient (bounded-state layers) also provides the perfect temporal window to move data. While the GPU is busy with a Mamba layer, the NVLink-C2C interconnect is free to fetch the next Attention layer's prefix.
  • Transparency is King: Previous dynamic sharing schemes often required rewriting attention kernels or complex pointer management. By operating at the virtual memory page-table level, TideCache remains "transparent" to the rest of the software stack (like SGLang), making it much easier to integrate into production systems.
  • Streaming as Capacity Expansion: Streaming isn't just a fallback for when you run out of memory; it effectively expands the "virtual" size of the GPU's HBM. The authors show that streaming can increase effective global-KV capacity by ~77% for large models.
Figure 1: Two opportunities for efficient hybrid-model serving: sharing cache capacity and exploiting idle decode-time bandwidth. years (Table 1): from 64 GB/s over PCIe 4.0 on A100 to 900 GB/s over NVLink-C2C on Grace Hopper, and 1.8 TB/s on Vera Rubin (Evans et al., 2022; Aubrey & Stam, 2025; Aubrey, 2026). This allows GPUs to access host-resident data through both direct memory accesses and explicit trans- fers. Host memory thus provides an additional tier for the different cache components maintained by hybrid models.

Key Results

  • Throughput: TideCache completes up to 2.04× as many agent turns compared to SGLang with HiCache (a state-of-the-art offloading baseline).
  • Latency: It reduces the median (P50) per-token latency by 20–59% and improves the P50 Time-To-First-Token (TTFT) for agents by 1.40–3.05×.
  • Adaptability: In shifting workloads (moving from chat bursts to agent bursts), TideCache's dynamic repartitioning allowed it to complete 59% more turns than static baselines, which hit a "memory wall" and were forced to queue requests.

Limitations and Future Work

  • Hardware Specificity: TideCache is heavily optimized for "superchips" with high-bandwidth CPU-GPU interconnects (like NVLink-C2C). Its benefits would likely be significantly diminished on standard PCIe-based systems where the "compute gap" isn't wide enough to hide the transfer latency.
  • Prefill vs. Decode: While the paper focuses heavily on the decode phase, the authors note that prefill computation still occupies the GPU and can delay subsequent requests. Future work could look at better interleaving of prefill and streamed decode tasks.
  • Kernel Integration: While the system is "transparent," the authors suggest that even further gains could be found by co-designing attention kernels that are aware of the streaming slots to reduce memory fragmentation.

Summary updated

Abstract

Hybrid architectures have become mainstream among open-weight language models. Their global-attention and bounded-state layers maintain heterogeneous caches whose demands vary with context length and concurrency. We present TideCache, a serving system on CPU--GPU superchips with three components for allocating GPU memory among caches and managing their residency across GPU and host memory. The Transparent Cache Pool redistributes physical pages among cache components behind fixed virtual addresses, preserving cache layouts and backend interfaces. The Cache Streamer uses computation intervals provided by bounded-state layers to prefetch the next global-attention layer's prefix into GPU slots shared across global-attention layers, reducing GPU residency while overlapping layer-wise transfers with computation. Generated KV remains resident. The Request Scheduler selects resident or streamed execution and triggers capacity adjustments according to request size and queue state. Together, they let HBM released by streaming accommodate other cache components and additional requests. Implemented in SGLang and evaluated on GB300 and GH200 across seven model/platform configurations, TideCache completes up to 2.04x as many agent turns, reduces overall P50 per-token latency by 20--59%, and improves agent P50 TTFT by 1.40--3.05x compared with SGLang with HiCache.
BibTeX
@misc{luo2026tidecache,
  author = {Jinbin Luo and Xiaozhe Yao and Shang Yang and Zhijian Liu and Song Han and Qinghao Hu},
  title = {{TideCache: Efficient Hybrid LLM Serving with Transparent Cache Management and Layer-Wise Streaming}},
  year = {2026},
  eprint = {2611.26619},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26619},
}
Figure