When a language model chats with someone over hundreds of turns, it stores a running memory of every past token: the KV cache, the saved key and value vectors that let attention look back without recomputing. Serving systems such as vLLM keep this memory in fixed 16-token pages, and compression methods decide which pages to keep on the GPU and which to drop or move to CPU memory. These decisions are made page by page or token by token, while conversations are organized into topics: a user states a constraint, qualifies it two turns later, and revisits it an hour after that. This note proposes making the topic episode the unit that the cache evicts and restores. The cache stays physically paged, but it also keeps a lightweight table that groups pages into topic segments. Boundaries between segments are detected from the model's own key vectors, so no extra model has to run. When a new turn arrives, whole episodes are scored and brought back together.
The bet is specific. In dialogue, a new question tends to attend strongly to only part of the material it depends on. Selecting individual pages therefore recovers the "needle" but loses the qualifications around it, and promoting the surrounding episode should repair this at the same memory budget. The note proposes testing that premise before building anything, using an LLM judge in a counterfactual role. The judge's job is to determine which past segments a correct answer actually needed, and to score the cross-turn contradictions that current KV-compression benchmarks never measure.
Where this sits
Three lines of work supply the pieces, and each stops short in a different place. Paging comes from PagedAttention, which maps logical blocks to non-contiguous physical blocks and keeps memory waste under 4%. Query-aware page selection comes from QUEST. QUEST keeps min-max bounds on each page's keys, estimates an upper bound on attention, and loads the top pages; it reports 100% passkey accuracy at a 1% budget, where H2O and TOVA fell to 0–4%. Reversible eviction comes from Cache You Later (RepairKV). RepairKV spills evicted tokens to host memory and promotes them back when the next turn arrives, using a fixed burst of 2 tokens before and 20 after each anchor. All three units of recall are set by token offsets rather than discourse. RepairKV also says explicitly that it has not been adapted to page tables. This proposal reuses QUEST's per-page metadata and RepairKV's offload-and-promote loop, and changes only the unit that gets promoted.
The closest prior work is EpiCache (and its QA version). EpiCache clusters the conversation offline into episodes using a sentence encoder and builds a compressed cache per episode. At query time it routes each query to one episode's cache, and it reports gains of up to 20 points over SnapKV and KVzip at 2K–4K budgets on Realtalk. It differs from this proposal in three ways. Its episodes are fixed by an offline clustering pass and do not follow a live conversation. Its caches are per-cluster copies rather than segments laid over a shared page table. Its routing is one-of-K, so a query that needs two episodes is poorly served. Here segmentation is online and works on the cache itself, eviction is reversible, and several segments can be promoted at once.
The segmentation literature motivates the premise and constrains the detector. Topic shifts are real and matter downstream: Topic-Aware Multi-turn Dialogue Modeling found boundaries at "similarity valleys" between adjacent windows, and weighting segments gave a 3.3% R10@1 gain on a topic-shifting E-commerce corpus. Dialogue acts mark segment ends: SuperDialseg reports a 75.2% chance that a segment ends after an agent's "providing a solution" act. Unsupervised detectors are weak, however. CSM reaches F1 0.381 against 0.784 for supervised RoBERTa on SuperDialseg, and summarization-based denoising (SumSeg) is too slow for serving at up to about 19 seconds per document. A key-state detector therefore has to show it is not simply a weak unsupervised segmenter.
The judge side supplies the measurement. VISTA breaks assistant turns into atomic claims, resolves cross-turn references, and verifies the claims against accumulated history, reaching Krippendorff's α = 0.832 with humans. AgentEval shows that end-to-end scoring misses most intermediate failures (recall 0.41 versus 0.89 for step-level judging). The proposal borrows that lesson for cache compression, where only final QA accuracy is usually reported. LLM-as-a-Judge on a Budget supplies ROBIN-HOOD, a variance-aware way of allocating repeated judge calls. The judge here is an evaluation instrument, not a component of the serving path. Despite its name, Judge Q is a learned-query eviction method, and it is treated as a compression baseline rather than as judging work.
Core Claim
At equal GPU memory, promoting whole topic segments recovers cross-turn dependencies that fixed-page and fixed-burst selection miss. This shows up as fewer contradictions of earlier user-stated facts and higher long-conversation QA accuracy. The boundaries needed can be found from key states the cache already holds, at negligible cost.
The claim fails if either of two things happens. The first is that query-aware page selection at the same budget closes the gap. The second is that randomly placed segments with the same length distribution do just as well, which would mean any contiguity helps and topic structure adds nothing.
Research Program
Stage 0: test the premise with counterfactual segment attribution. Take long histories from LoCoMo, Realtalk, and LongMemEval. Generate the answer or next assistant turn with the full cache and with a compressed cache, using SnapKV and QUEST at a 2K budget. Where the compressed output is wrong or contradicts history, restore one reference segment at a time, using human-annotated or RoBERTa-predicted boundaries, and regenerate. A VISTA-style claim-level judge decides whether each restoration fixes the failure.
This produces, for each failure, the set of segments it needed. Two measurements follow. The first is how often a failure needs pages from a segment that the query-aware selector partly loaded but did not load in full. The second is the dependency concentration ratio: the fraction of the needed tokens that lie inside segments the selector already touched.
If the needed tokens are usually isolated pages, or scattered across untouched segments, the core claim is wrong. In that case the work stops at a diagnostic contribution: a judge-based failure attribution for compressed caches, which is the step-level evaluation AgentEval argues for, applied to memory.
Stage 1: segment boundaries from key states. Each turn is summarized by the mean of its key vectors over a few middle layers. This is computed during prefill or decoding at no extra forward-pass cost, alongside QUEST's per-page min-max metadata. A boundary is placed at a valley in cosine similarity between adjacent turn windows, using the similarity-valley rule from TADAM, with Savitzky–Golay smoothing borrowed from SumSeg to suppress backchannel noise. Boundaries snap to page edges. The page straddling a boundary belongs to both segments, so physical layout and vLLM's waste figure stay unchanged; the actual overhead will be measured. Boundaries are updated incrementally as turns stream in. Detector quality is scored on SuperDialseg, Dialseg711, and TIAGE before it is used for any eviction.
Stage 2: segment-granular eviction and repair. Pages beyond the budget are offloaded to host memory, following RepairKV. At each new turn, a segment's score is the maximum of the QUEST upper bounds over its pages. The policy promotes whole segments in score order until the budget is filled and evicts the lowest-scoring resident segments to make room. A segment larger than the remaining budget is truncated to its highest-scoring pages plus its first turn. The policy is implemented over a vLLM-style block table, which also addresses the integration gap RepairKV leaves open.
Stage 3: judged conversational evaluation. Beyond QA accuracy, the evaluation generates continuation turns on held-out histories. It scores contradicted claims per turn and lacking-evidence claims per turn using VISTA's categories, with history verified against the full transcript. The judge is validated on a 300-turn human-annotated subset, with α reported against VISTA's 0.832 as a reference point. Repeated judge samples are allocated with ROBIN-HOOD.
Baselines
The naive reference is recency eviction in the style of StreamingLLM, which keeps attention-sink tokens plus the most recent window. It needs no metadata and represents what a serving engine does by default; matching it would mean the work adds nothing. Full uncompressed KV is the ceiling.
The strongest existing competitor on this exact question is RepairKV-style reversible eviction with burst expansion, because it already solves the multi-turn relevance shift that motivates this work. On Qwen2.5-7B at 32K context, 4-query MQ-NIAH, it reports 91.0% retrieval accuracy versus 24.5% without repair, promoting only 96 tokens. It will be run with identical offload storage, the same promotion budget in tokens, and the same idle-window timing. It will also be given a tuned burst width, sweeping the after-anchor offset from 20 to 200 tokens with a budget for this sweep equal to the one spent tuning the segment detector. This answers the obvious objection that a wider burst is all that is needed. Beating it means higher contradiction-free rate and QA accuracy on LoCoMo and Realtalk at matched budget, while staying within two points of its MQ-NIAH accuracy, where segment structure should not help.
QUEST with fixed 16-token pages is the strongest query-aware selector. It is run on the same model with the same page metadata, the same dense early layers, and the same token budget, so the only difference is page-wise versus segment-wise selection. QUEST reports parity with full cache on LongBench; no reported result found on Lacuna for QUEST on LoCoMo, Realtalk, or LongMemEval.
EpiCache is the strongest topic-based method. It is run on its own benchmarks with its released configuration at 2K and 4K budgets. Its paper reports up to 20-point gains over SnapKV and KVzip on Realtalk but no absolute figure in the file, so its numbers will be reproduced rather than quoted. SnapKV and Judge Q serve as additional one-way eviction references.
Two ablations isolate the key ingredient. The first replaces detected boundaries with random boundaries drawn to match the segment-length distribution. The second uses fixed-length segments of the mean detected length. A third ablation substitutes boundaries from a RoBERTa segmenter trained on SuperDialseg, which tests whether key-state boundaries lose anything against a text model.
For the segmentation stage alone, the bar is the reported RoBERTa F1 of 0.784 in domain and 0.660 out of domain on Dialseg711, with CSM's 0.381 as the unsupervised floor. The key-state detector does not need to beat supervised RoBERTa. If it falls near CSM, though, the RoBERTa-boundary ablation becomes the system to ship.
Why not just…?
Why not just use QUEST's page selection, which already picks whatever pages the query needs? This is the real threat. If needed context always draws high attention from the new query, page-level top-K already finds it, and segments add only overhead. The premise is that qualifying material, such as "but not on weekdays" said two turns after the constraint, receives low attention from a later query about the constraint. Stage 0 measures exactly how often that happens before anything is built, and the matched-budget QUEST comparison settles it. If the dependency concentration ratio is low or QUEST ties, the claim is dropped.
Why not just widen RepairKV's burst window? A wider burst is contiguity without topic structure. It will over-promote across topic boundaries when the anchor sits near the end of an episode, and under-promote when the qualification lies before the anchor, since the burst extends only 2 tokens backward. The tuned-burst baseline and the fixed-length-segment ablation together show whether topic-aligned boundaries matter beyond simple span width.
Why not just rerun EpiCache's clustering periodically? Re-clustering a growing history means re-embedding it and rebuilding per-cluster compressed caches. That is the offline cost EpiCache's design accepts. Its one-cluster routing also cannot serve a query that spans two episodes. The comparison will include the slice of LoCoMo and Realtalk questions whose Stage 0 attribution needs segments from more than one episode, where routing should fail and multi-segment promotion should not.
Why not just retrieve raw transcript text and re-prefill it? RepairKV reports re-prefill of 32K context at over 2 seconds against about 110 ms for cache-level repair, and EpiCache shows the peak-memory spikes that long re-prefills cause. Text retrieval will still be included as a latency-accuracy point, because at small histories it may win.
Decisive Checks
The experiments use Qwen2.5-7B and LLaMA-3.2-3B, matching EpiCache, at 2K, 4K, and 8K budgets on LoCoMo, Realtalk, LongMemEval, and MQ-NIAH, with 32K–100K histories.
Three results decide the outcome. The first is the Stage 0 dependency concentration ratio, which says whether the premise holds. The second is the head-to-head against QUEST and tuned-burst RepairKV on judged contradictions and QA accuracy at matched budget, with the random-boundary ablation as the control that separates topic structure from mere contiguity. The third is overhead: detector cost per turn, block-table bookkeeping, and change in memory waste relative to PagedAttention's 4%, all reported against end-to-end decode latency.
A positive outcome needs all three. That means a substantial share of failures recoverable within partly loaded segments, a gain over both strong baselines that disappears with random boundaries, and overhead under a few percent of decode time. Anything less still yields a judge-based tool for explaining why compressed caches fail in conversation, which the field currently lacks.
References
- Efficient Memory Management for Large Language Model Serving with PagedAttention
- QUEST: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Cache You Later: Post-Compression KV Repair for Long-Context Agentic LLM Inference
- EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
- EpiCache: Episodic KV Cache Management for Long Conversational Question Answering
- Topic-Aware Multi-turn Dialogue Modeling
- SuperDialseg: A Large-scale Dataset for Supervised Dialogue Segmentation
- Leveraging Summarization for Unsupervised Dialogue Topic Segmentation
- VISTA Score: Verification In Sequential Turn-based Assessment
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- LLM-as-a-Judge on a Budget
- Judge Q: Trainable Queries for Optimized Information Retention in KV Cache Eviction