Hop-Scheduled KV Eviction for Multi-Hop Reasoning Inside Hardware Enclaves

A language model that answers a multi-hop question, such as "in what city was the director of Inception born?", has to look something up, reason about it, and then look up something else. The standard way to do this, interleaving retrieval with step-by-step reasoning, keeps adding retrieved passages and reasoning sentences to the prompt. Its working memory therefore grows with every hop. That working memory is the KV cache: the stored key and value vectors of every past token, which the model reads at each generation step. Now suppose the whole computation has to run confidentially inside a trusted execution environment (TEE), a hardware-isolated region whose memory the host operating system cannot read. Two problems appear together. Protected memory is scarce and expensive to spill. And when the cache does spill, the untrusted host can watch which memory pages are evicted and brought back, even though it cannot read their contents.

We propose to manage the KV cache one hop at a time rather than one token at a time. The question, the system instructions, and each hop's short generated reasoning sentence stay resident. Each hop's retrieved passages sit in fixed-size, page-aligned blocks and are dropped as whole blocks once the hop's reasoning sentence has been written. The retained sentence usually names the bridge entity, the intermediate fact that links one hop to the next. Because this schedule depends only on the hop counter and never on attention scores, the page trace the host sees carries no information about which passage mattered. The policy also needs no per-token scoring inside the enclave. We will test whether this simple schedule matches token-level compression on answer accuracy, beats it on real-silicon cost, and removes a leakage channel that attention-driven eviction opens. Every cost is measured on physical TEE hardware as well as in the instruction-count simulators that prior confidential-ML work relied on.

Core Claim

We make three claims about a model serving interleaved retrieval-and-reasoning pipelines inside an enclave. First, hop-granular, content-independent eviction reaches answer F1 on MuSiQue, HotpotQA, and 2WikiMultihopQA within a small margin of the best token-level compression methods at equal memory budget. Second, it has lower wall-clock latency and energy per question on real enclave hardware, because it skips the per-token scoring kernels and the fragmented page movement those methods cause. Third, the page-access trace it produces reveals the supporting evidence no better than chance, while attention-driven eviction reveals it well above chance.

The third claim is new. Existing eviction papers treat compression purely as a quality-versus-memory trade-off. The Pitfalls of KV Cache Compression shows that eviction has security consequences, since it unevenly drops system-prompt defenses. But that paper considers only what the model forgets, not what the eviction pattern tells an observer. We add a fourth, methodological claim: the ranking of eviction policies by cost changes between instruction-count simulation and physical silicon. If that holds, simulator-only evaluation of confidential inference is misleading for exactly this workload.

Where This Sits

The closest confidential-LLM work is Towards Confidential and Efficient LLM Inference with Dual Privacy Protection. It runs only the embedding layer inside Intel SGX2 and sends noise-perturbed tokens to an untrusted GPU, so the KV cache is never protected. It is also evaluated only on single-step tasks. DarkneTZ partitions CNNs across Arm TrustZone on real edge hardware, and Marshaled Learning splits vision networks across memory-limited confidential VMs. Both confront enclave memory limits, but for static feedforward activations rather than a cache that grows during decoding. GuaranTEE runs a whole model inside an Arm CCA realm, a hardware-isolated virtual machine. It reports a 1.7x instruction overhead for MobileNetV1 inference, measured on the Arm Fixed Virtual Platform simulator, and it states that the simulator misses memory-bus and cache contention. These papers motivate our setting. We differ in putting the entire autoregressive state inside the protected boundary and in measuring on silicon.

On the reasoning side, IRCoT alternates one reasoning sentence with one retrieval call. It reports 11–21 point retrieval recall gains and up to 15 F1 points over one-shot retrieval. Its design is the source of the linear context growth we manage, and we reuse its pipeline unchanged. MuSiQue supplies connected questions without shortcuts. HotpotQA supplies gold supporting-fact labels, which we need to score leakage.

On the memory side, four methods set the quality bar. R-KV prunes redundant reasoning tokens by combining attention and key similarity. BeaconKV protects early planning tokens that models later "revisit". DefensiveKV tracks worst-case rather than mean importance. DuoAttention keeps full history only for about 25% of heads. All four were built and measured on unprotected GPUs, and all make eviction decisions that depend on attention values.

Two of these findings constrain our design. BeaconKV's "thought revisiting" and DefensiveKV's non-stationary importance spikes warn that anything evicted may later be needed. Our bet is that in multi-hop retrieval the reasoning sentence absorbs what later hops need from a passage. That is an empirical claim, and experiment one tests it directly. R-KV's authors also note that integrating dynamic eviction with paged memory is unresolved. Hop-granular, page-aligned blocks sidestep that problem by construction.

Research Program

We will build an enclave-resident inference runtime for a CPU-served model. The runtime holds model weights, the dense retriever, its index, and the KV cache inside the protected boundary. It runs on three substrates: Intel SGX2, as used in the dual-privacy work; a commercial confidential VM of the kind used by Marshaled Learning; and an Arm CCA realm on the Fixed Virtual Platform, following GuaranTEE's setup. Physical CCA silicon will be added if we can obtain it. The corpus is public Wikipedia, and the retriever scans the full index for every query, so the retrieval step touches every vector. The residual leakage we study is therefore the KV paging pattern and passage fetches. We model the threat as a host that sees page-granular eviction and fetch events but not their contents.

The hop-scheduled policy works as follows. Every retrieved passage is padded or truncated to a fixed token length, so each hop occupies the same number of cache pages. The question and system prompt, together with a small attention-sink prefix, are pinned. All generated reasoning sentences are retained. Passage blocks from hop $t$ are evicted when hop $t+1$ begins, apart from an optional window of the most recent $w$ hops. Retained tokens keep their original positions. The schedule is a fixed function of the hop index and $w$, which is the property the leakage claim rests on. We choose $w$ on a held-out development split, once per budget.

We also build an instrumented tracer that records page events from inside our own runtime; this stands in for the host's view. We attach a cost harness that reports instruction counts in simulation and wall-clock latency plus package energy on physical machines. Accurate and stable empirical CPU power modelling shows that energy estimated by weighting event counts misses non-linear effects, so we measure energy directly rather than infer it from counts.

Baselines

The naive reference points are two. The first is full-cache IRCoT inside the enclave, with no compression; where the cache exceeds protected memory, this measures paging cost directly. The second is one-shot retrieval with the same total passage budget, which anchors how much multi-hop interleaving is worth under enclave constraints. For quality, the target is IRCoT's own result. The file reports IRCoT gains only in aggregate, as 11–21 recall points and up to 15 F1 over one-step retrieval, with Flan-T5-XL plus IRCoT beating GPT-3 with one-step retrieval. It gives no per-dataset absolute F1, so we will reproduce full-cache IRCoT ourselves and treat that as the ceiling.

The strongest existing competitors are token-level eviction policies run inside the same enclave at the same budget. The main one is DefensiveKV, whose paper reports 2.3x to 4.3x less quality loss than CriticalKV at a 20% cache budget on LongBench, and near-lossless performance at 40% cache on Llama-3.1-8B. R-KV and BeaconKV are included because they were designed for long reasoning traces. R-KV reports matching full-cache accuracy with 10–34% of the cache on math reasoning, and BeaconKV reports a 31.7-point margin on AIME24. DuoAttention is included as a head-level alternative. We found no reported result on Lacuna for any of these methods on MuSiQue, HotpotQA, or 2WikiMultihopQA, so all numbers on those benchmarks will be produced under our harness.

Fairness rules are as follows. Every method uses the same model, the same retriever, the same pipeline, and the same KV budget, stated in bytes of protected memory. Every method gets the same development-split tuning budget for its hyperparameters. Every method runs its own scoring kernels inside the enclave, so their cost is counted.

To beat DefensiveKV, the hop schedule must stay within 2 F1 points of it at the same budget. It must also be faster in wall-clock time and lower in energy per question on physical hardware, and it must show no significant leakage where DefensiveKV shows some.

Two ablations isolate the key ingredient. The first is hop-granular eviction whose retained passages are chosen by attention mass rather than by schedule. It keeps page alignment but restores content dependence, which separates the leakage benefit from the granularity benefit. The second is the fixed schedule without passage padding, which tests whether variable lengths alone leak.

Why not just…?

Why not just drop passages from the prompt text and re-encode the shorter prompt each hop? This is the most serious threat. Rewriting the context at hop boundaries and re-processing it is almost the same policy as ours, and it is simpler. It pays a full re-prefill of the retained context at every hop, whereas we pay none. Our expectation is that the difference grows with the number of hops and the length of the reasoning sentences, and is negligible for two-hop questions. We include it as a full baseline. If it matches our cost on silicon across MuSiQue's 2–4 hop questions, we will report that the right recommendation is text compaction and that our contribution reduces to the leakage and measurement findings.

Why not just use a VM-level enclave with gigabytes of protected memory and skip compression? Confidential VMs and CCA realms do relax the tight limits that shaped DarkneTZ. The full-cache baseline measures exactly this option. BeaconKV reports over 77GB of cache for a 4B model generating 32K tokens, and multi-hop pipelines stack retrieved passages on top of that. We therefore expect encrypted-memory bandwidth, not capacity alone, to dominate decoding cost. That expectation is a hypothesis, not a cited result. If the full-cache configuration is within noise of the compressed ones on latency and energy, the memory argument fails, and only the leakage argument remains.

Why not just run DefensiveKV or R-KV inside the enclave? On accuracy they may well win, because they were designed for non-stationary importance. Our argument against them is not quality. Their eviction choices are functions of attention scores, so the page trace encodes which content the model found relevant. Their scoring kernels also add memory traffic, which is costlier under encryption. The leakage experiment and the silicon cost comparison settle both points directly.

Why not just add noise to embeddings on the client instead of using an enclave? The dual-privacy work shows that token-substitution noise preserves utility on classification and summarization. Multi-hop retrieval, however, needs an exact bridge entity to fetch the next document, and MuSiQue removes the shortcuts that might tolerate corrupted entities. A noisy-embedding IRCoT run on MuSiQue will show whether retrieval recall survives. We report it as a side result rather than a main baseline, because it gives a different privacy guarantee.

Proposed Experiments

The first check is quality. We run IRCoT with Flan-T5-XL, following the original paper, and with an 8B decoder model, on MuSiQue-Ans, HotpotQA open-domain, 2WikiMultihopQA, and IIRC. Each policy is run at cache budgets of 10%, 20%, and 40% of full, and we report answer F1 and retrieval recall. The hop-bottleneck hypothesis predicts that errors under our policy concentrate on questions whose reasoning sentence omits the bridge entity. We will annotate a sample of such questions to confirm or refute this.

The second check is leakage. We use the HotpotQA distractor setting, which places 2 gold paragraphs among 8 distractors, and run it inside the enclave. For each policy we collect page-event traces and train a classifier to predict the gold paragraphs from the trace alone, then report accuracy against the chance rate. The decisive outcome is attention-driven policies clearly above chance and the hop schedule at chance. If the hop schedule also leaks, for example through timing, the leakage claim is falsified.

The third check is the simulation-to-silicon gap. For every policy we record instruction counts on the Fixed Virtual Platform and wall-clock latency and energy on SGX2 and a confidential VM, then compute the rank correlation between the simulated and physical orderings of policies. Air Learning found a 37.73% flight-time gap between desktop simulation and embedded hardware, which motivates expecting disagreement. Our specific claim is that token-level scoring looks cheap when counted in instructions but expensive once memory traffic is counted. If the rankings agree, we will report that instruction-count simulation is adequate for this workload, which is also a useful result.