Retrieval-augmented generation (RAG) systems look up text passages, or "chunks," and feed them to a language model. Recent serving systems save compute by caching each chunk's precomputed attention state, its KV cache, and reusing it across requests. The question is which chunks to keep. Lifetime-based caching says to keep the items whose next request is expected soonest and to evict the ones that have effectively expired. That rule needs a good estimate of each item's request timing, and the field has flagged this estimate as its weak point. RAG traffic makes it worse because of vocabulary mismatch: users and query rewriters phrase the same need in many ways. Different phrasings retrieve different but nearly identical chunks. The requests for one piece of information therefore scatter across several chunk IDs. Each chunk's request history is thin and noisy, and the cache stores several copies of the same content, each valued separately.
The proposal is to treat a group of near-duplicate chunks as one cache unit with one shared request history. Lifetimes are then estimated from the combined history of the group. When a request asks for a chunk that is not cached, the cache can serve a resident member of the same group instead. Redundancy, normally a nuisance, becomes a source of statistical strength: pooling over a redundancy group turns many sparse request processes into one dense one. Vocabulary mismatch is the reason this pooling is needed. The intended result is fewer chunk recomputations at equal cache size, with answer quality checked by a coverage metric that is sensitive to lost information.
Core Claim
Under paraphrase-heavy RAG traffic, a cache that estimates remaining lifetime per redundancy group should beat per-chunk frequency and time-to-live (TTL) policies on hit ratio and prefill compute at fixed memory. Here a redundancy group is a cluster of chunks whose content overlaps above a threshold, and the group's lifetime comes from the pooled inter-request delays of all members. Serving a cached group member in place of a missing one should cost no measurable loss in sub-question coverage when the threshold is set by coverage rather than by raw similarity.
Each ingredient comes from prior work, and each prior system is missing the other two. Cache-Craft supplies the mechanism we reuse. It stores chunk caches with positional encodings removed so they can be reused at any position, and it selectively recomputes context-sensitive tokens. It keys caches by chunk identity and evicts by access frequency and storage tier, so it keeps every paraphrase-induced duplicate.
The lifetime lineage motivates the eviction rule. LISO replaces the cached item with the shortest remaining lifetime. Time Domain Buffer Sharing allocates how long items stay cached using the distribution of delays between an item's arrival and its request, called request delay information (RDI). It reports 36.4% to 51.8% storage savings at equal hit ratio. Continuum sets per-turn TTLs for agent KV caches. All three assume known or easily estimated delay distributions over independent, atomic items. Our contribution targets exactly that assumption.
TwinKV motivates measuring redundancy directly rather than through attention. It shows that attention magnitude has near-zero correlation with token utility (Spearman ρ = −0.004) and that 37–47% of long-context tokens have near-duplicates. It works within one sequence at a single moment, whereas we work across requests over time. This also bears on a tension in the file: Cache-Craft scores tokens with attention while TwinKV argues against attention. We take TwinKV's side and form groups from content similarity, not attention.
Research Program
Group construction. We build redundancy groups offline over the corpus and update them incrementally. Two chunks join a group when their embedding similarity exceeds a threshold θ and an answerability check agrees. The check asks whether one chunk answers the sub-questions the other answers, using CRUX-style sub-question coverage. We calibrate θ on held-out queries so that substitution keeps coverage within a fixed tolerance. The coverage check guards against a failure TwinKV documents: text that looks redundant can carry structure the model needs.
Pooled lifetime estimation. Each group $c$ keeps one record of inter-request delays, merged across all of its members. Its eviction value at time $t$ follows the break-even logic that InferMax adapts from database buffer pools:
$$v_c(t) = \frac{\hat{P}_c(\text{request within } \tau \mid \Delta_c(t)) \cdot R_c}{s_c}.$$
In this formula, $\Delta_c(t)$ is the time since any member of group $c$ was last requested. $\hat{P}_c$ is the probability of a new request within horizon $\tau$, estimated from the group's pooled delay history with a simple discrete hazard model. $R_c$ is the measured cost of recomputing the resident member's prefill. $s_c$ is its memory size, which handles variable-size items, another limitation the file notes. The cache evicts the lowest $v_c$, which is LISO's shortest-remaining-lifetime rule weighted by cost per byte.
Pooling should help most where per-chunk histories are sparse. The variance of a delay-distribution estimate shrinks roughly with the number of observed delays. Merging $m$ equally popular members multiplies that count by about $m$. We will check this prediction directly by measuring how estimation error changes as group size grows.
Substitution. On a miss for chunk $d$, if any member of $d$'s group is resident, the cache serves that member through Cache-Craft's reuse path. The recompute cost of the requested chunk is avoided, and the substitution is logged for quality auditing.
Workload. No public trace has paraphrase-driven chunk reuse, so we build one. We start from questions in the multi-hop sets Cache-Craft evaluates on (2WikiMQA, MuSiQue) plus HotpotQA. We expand each question into lexical variants with a RaFe-style rewriter and an InteR-style generator of expanded queries. Arrival times follow Zipf-distributed popularity with phase shifts that change which topics are hot. For long-form evaluation we use CRUX-DUC and CRUX-Multi-News, where coverage is defined. We vary the paraphrase fan-out per need from one to many. This sweep is the main experimental variable, because the claim predicts that gains grow with fan-out and vanish at fan-out one.
Baselines
The naive reference is prefix caching with least-recently-used eviction, the default in common serving engines. It sets the floor for hit ratio and compute.
The strongest existing method is Cache-Craft with its own frequency- and tier-based eviction. It reports 51% less redundant computation than prefix caching, 75% less than full recomputation, 2x lower latency and 1.6x higher throughput on LLaMA-3-70B, while keeping 90% of ROUGE F1. It is the only system in the file that caches retrieved chunks for reuse across queries. Our method runs on top of the same chunk-cache mechanism, so the comparison changes only the eviction and lookup policy. Both policies get the same model, corpus, retriever, GPU/CPU memory budget and workload trace. Both get the same tuning budget over their hyperparameters: Cache-Craft's recompute fraction and tier thresholds, and our θ and τ. Beating it means at least 15% less prefill compute at equal memory under fan-out of three or more, with CRUX coverage and downstream exact match not statistically worse.
The second strong baseline is per-chunk lifetime eviction: the same $v$ score computed from each chunk's own delay history, with TTL-style expiry in the spirit of Continuum. It isolates the value of lifetime modeling without pooling.
Two ablations isolate the key ingredient. Pooling-only uses group histories for eviction but never substitutes. Substitution-only serves group members on a miss but estimates lifetimes per chunk. If pooling-only captures most of the gain, the statistical argument holds. If substitution-only does, the contribution is really deduplication, and we will say so.
Why not just…?
Why not just run MMR at retrieval time and keep a standard cache? Maximal Marginal Relevance (MMR) reranks results to penalize similarity to passages already selected. It removes redundancy within one request's context, but the redundancy here lies across requests. Paraphrase one retrieves chunk A, paraphrase two retrieves its near-twin A′, and each context is individually diverse. CRUX also reports that even setwise rerankers stay well below oracle coverage, so retrieval-time diversity does not fix the problem on its own. We include MMR-before-retrieval plus Cache-Craft as an extra arm. The fan-out sweep should show it matching our method at fan-out one and falling behind as fan-out grows.
Why not just normalize queries upstream with a rewriter so paraphrases retrieve the same chunks? This is the most serious threat. If a RaFe-trained rewriter maps lexical variants to one canonical query, per-chunk histories become dense without any grouping. The file gives reasons for doubt: RaFe inherits the blind spots of the reranker it was trained against, and expansion methods like InteR deliberately fan queries out rather than collapse them. That is an argument, not evidence. We will run a canonicalizing rewriter in front of per-chunk lifetime eviction as a full arm with matched compute, including the rewriter's own cost. If it reaches our hit ratio, we will conclude that the gap is better closed upstream.
Why not just make the cache bigger? Extra memory raises every policy's hit ratio and hides policy differences. We therefore report full hit-ratio-versus-memory curves and the memory each policy needs to reach a target hit ratio. That is the same comparison under which time-domain buffer sharing reported 36.4–51.8% savings, and the claim must hold across the curve.
Proposed Experiments
The decisive experiment is the fan-out sweep on the synthetic multi-hop trace. We plot prefill compute and hit ratio against fan-out for all arms at three memory budgets. The claim predicts that the advantage of pooled-lifetime eviction over per-chunk eviction and over Cache-Craft grows with fan-out. If the curves stay flat, the idea fails.
The second check is estimator quality. We measure the log-loss of predicted next-request probability for groups against singletons, binned by group size. This tests whether pooling actually reduces lifetime-estimation error, the open problem it targets.
The third check is quality. We measure CRUX coverage on DUC and Multi-News, exact match on HotpotQA and MuSiQue, and an audit of logged substitutions. For context on the exact-match metric, Retriever Portfolios reports 0.464 against 0.356 for nearest-neighbor retrieval on Gemma-3-27B-IT. We do not expect to move that retrieval-side number, only to avoid degrading whatever the base pipeline achieves.
The fourth check is robustness under drift. We shift topic popularity mid-trace and measure how quickly each policy recovers its hit ratio. This addresses the sensitivity to sudden workload changes that the file notes for Continuum. For TwinKV-style within-sequence redundancy numbers in this setting there is no reported result found on Lacuna, so we do not treat that line as a numeric baseline.
References
- Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented Generation
- Energy-efficient wireless content delivery with proactive caching
- Caching With Time Domain Buffer Sharing
- Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Live
- TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy
- Controlled Retrieval-augmented Context Evaluation for Long-form RAG
- Faster LLM Inference using DBMS-Inspired Preemption and Cache Replacement Policies
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Synergistic Interplay between Search and Large Language Models for Information Retrieval