Large language models answering multi-hop questions, such as "in what city was the director of Inception born?", hit two walls together. The first is about reasoning. The original question does not mention the bridge fact ("Christopher Nolan"), so anything that picks evidence using only the question will miss the second hop. The second is about memory. The pool of candidate evidence is far too large to keep in fast GPU memory as precomputed attention state, so most of it has to live in slower CPU memory.
Systems that move attention state between the two memories already exist. They decide what to bring back to the GPU by comparing stored tokens against the incoming user query. One of the strongest such systems found that a stale query, one written before the needed fact became relevant, gives almost no benefit. We propose to use the model's own step-by-step explanation as the fresh query. Each time the model writes a reasoning sentence that names a new entity, that sentence, or the hidden state that produces it, chooses which evidence blocks to copy back into GPU memory before the next sentence is written. The final answer is then a classification conditioned on the question, the explanation, and the evidence that the explanation pulled in.
The bet has three parts. Explanations close the multi-hop gap, as iterative retrieve-and-reason systems already show. They can do so at the level of the memory hierarchy rather than through repeated text retrieval and re-reading. And because a hop becomes identifiable partway through generating the rationale, the transfer between memories can overlap with decoding instead of relying on idle time.
Core Claim
Three terms first. The KV cache is the per-token key and value tensors a transformer stores so it does not recompute attention over earlier text. A memory tier is one level of storage: GPU memory (fast and small) or host CPU DRAM (slower and large). Promotion means copying a stored block of KV from CPU back to GPU.
The claim is this. In multi-hop question answering over an evidence pool whose KV cache does not fit on the GPU, scoring CPU-resident blocks against the model's in-progress rationale recovers most of the accuracy of iterative text retrieval with re-prefill, at a small fraction of its latency. Scoring the same blocks against the original question recovers little. A further claim is that a middle-layer hidden state, captured while the model is still writing the rationale sentence, is a good enough promotion signal to start the transfer early, so its cost is hidden behind decoding.
How This Sits Against Prior Work
Three groups of papers motivate the proposal by documenting the gap.
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT) alternates between writing one reasoning sentence and using it as a search query against a text corpus. It reports retrieval-recall gains of 11–21 points and question-answering F1 gains of up to 15 points. This establishes that rationales are the right bridge between hops. However, IRCoT re-reads every retrieved paragraph as fresh prompt text and keeps everything on the GPU. Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) shows that middle layers resolve the bridge entity before it is written out. It maps those hidden states through a small network into a dense retriever's space, with about 2.8 seconds per query on 2WikiMultiHopQA against over 20 seconds for iterative methods. L-RAG still fetches text and does not manage where attention state lives.
Cache You Later: Post-Compression KV Repair for Long-Context Agentic LLM Inference (RepairKV) supplies the closest method. It keeps evicted KV in CPU memory and scores it against a new turn's query. It promotes high-attention spans plus a surrounding window, and reaches 91.0% against 24.5% on a four-query needle task in about 110 ms. RepairKV both motivates and constrains this proposal. It is triggered by external turns, it hides transfer cost in tool-waiting idle windows, and it reports that stale queries fail. It never asks whether the model can generate its own fresh query within a single turn.
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory supplies the storage layout we reuse. It splits KV into blocks in CPU memory, each indexed by a few representative tokens, and loads blocks by relevance to the current token. Its relevance signal, however, is the current decoding token, with no notion of a hop. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation shows that precomputed document KV can be tiered across GPU and CPU memory, cutting time-to-first-token by 1.2–4× over vLLM. Its prefix tree assumes document order is known in advance, which is exactly what multi-hop reasoning violates.
On the classification side, Explain Yourself! Leveraging Language Models for Commonsense Reasoning (CAGE) shows that conditioning a classifier on a generated explanation lifts CommonsenseQA from 54.8% to 64.7%. Gold explanations reach about 90%, so the label is bottlenecked by explanation quality. We adopt its question-to-explanation-to-label structure. We change what the explanation does: it also fetches grounding evidence, which targets the quality bottleneck CAGE identified.
Two eviction papers constrain how we treat the rationale tokens themselves. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models shows that repetitive reasoning tokens can be dropped, keeping about 105% of full-cache AIME performance at 38% of the cache. Taming the Fragility of KV Cache Eviction in LLM Inference shows that tokens with low past attention can suddenly become critical. We do not take sides between them. We keep the rationale fully on the GPU by default and test both policies as an analysis.
Research Program
The system has four parts.
Offline encoding. Every passage in an evidence pool is encoded once, and its KV is stored in CPU DRAM as InfLLM-style blocks, each with a few representative key vectors held on the GPU as an index.
Generation and promotion. At inference, the GPU holds the question, the growing rationale, and a fixed budget of promoted blocks. The model writes its rationale one sentence at a time. At each sentence boundary, the queries from that sentence score the block index, and the top-scoring blocks plus neighbouring blocks are promoted. Neighbours are included in the spirit of RepairKV's burst expansion. Promoted blocks replace the lowest-scoring resident blocks.
Early-trigger variant. In a second variant, a small projection trained on supporting-fact supervision maps a middle-layer hidden state into the block-representative space, following the L-RAG recipe. Promotion can then be issued a few tokens into a sentence, once the projected score passes a confidence threshold. The transfer then runs in parallel with the remaining decoding.
Final answer. The answer, or label for yes/no and comparison questions, is produced conditioned on the question, the rationale, and the promoted evidence.
There is one technical risk that must be addressed directly. Passage KV computed in isolation and then spliced into a new context is not identical to KV computed in that context; RAGCache avoids this by exact prefix reuse. We handle positions the way InfLLM does, assigning promoted blocks positions relative to the local window. We then measure how much accuracy this approximation costs by comparing against re-prefilling the same chosen passages as text. If splicing alone loses more than a few F1 points, the design must add a short re-encoding pass over the promoted blocks, and we will report it as such.
Baselines
The naive reference point is single-shot promotion. The original question scores the block index once, the top blocks are promoted under the same GPU budget, and the model answers with an explanation. This is the stale-query condition that RepairKV reports as nearly useless, translated to multi-hop question answering. The proposal must beat it by a wide margin on hop-2 evidence recall; if it does not, the core premise fails. A second naive point is direct answering with no rationale, matching CAGE's 54.8%-style baseline in spirit. It shows how much the explanation contributes beyond fetching.
The strongest accuracy competitor is IRCoT. Each rationale sentence queries a dense retriever over the same passage pool, and the retrieved paragraphs are re-read as prompt text. The file gives no absolute IRCoT F1 on these benchmarks, only relative gains of up to 15 F1 points, so no reported result was found on Lacuna for an absolute IRCoT score. We will run it ourselves. IRCoT is the reference that answers "is the explanation-as-query idea worth anything without the memory machinery?"
The strongest low-latency competitor is L-RAG. Its reported figures, both from L-RAG, are 0.352 accuracy on MuSiQue, the highest in its comparison, and about 2.8 seconds per query on 2WikiMultiHopQA.
All three run on the same backbone model, retriever-equivalent passage pool, top-k budget, and GPU memory budget. All get the same hyperparameter search budget, and the L-RAG projection and ours receive the same training data.
Beating them means one of two things. The proposal could match IRCoT's F1 within two points at lower latency than L-RAG. Or it could exceed L-RAG's accuracy at comparable latency, while holding the same or less GPU memory.
The ablation that isolates the key ingredient is RepairKV-style repair driven by the question alone versus by the rationale, with everything else fixed. We also include a version that rescores with every decoded token, as InfLLM does, to separate the value of hop-level triggering from mere frequent rescoring.
Why not just…?
Why not just run IRCoT and re-read retrieved paragraphs? This is a real accuracy threat, and it may match or beat us on F1, because it has no KV-splicing approximation. Its cost is latency and memory. Every hop re-prefills new paragraphs, and L-RAG reports iterative loops exceeding 20 seconds per query against about 2.8 seconds for single-step methods. RepairKV measures re-prefill of 32K tokens at over 2 seconds against about 110 ms for repair. Our experiments report both axes. If IRCoT wins on accuracy by a large margin at a latency gap we cannot close, the conclusion is that splicing precomputed KV is not yet viable for multi-hop reasoning, and we will say so.
Why not just apply RepairKV at every sentence boundary? This is the most serious threat, because it is nearly our sentence-boundary variant. It is therefore a baseline, not a strawman. If it matches our text-rationale variant, then the contribution narrows to two things. The first is the empirical finding that a self-generated rationale removes RepairKV's stale-query failure. The second is whether the middle-layer early trigger buys real latency by overlapping transfer with decoding, which RepairKV cannot do without idle windows. We measure the second directly as end-to-end time per query with and without overlap.
Why not just hold everything on a large GPU? For distractor-setting HotpotQA, ten paragraphs fit easily, so on that setting this alternative wins and we will report it. The proposal targets pools of thousands of passages whose KV exceeds GPU memory. There, InfLLM's finding that unpruned long context distracts attention, and R-KV's finding that a compressed cache can outperform the full one, suggest that holding more is not automatically better. We test this with a full-context run at the largest pool that fits.
Why not just answer without an explanation? CAGE's 54.8% to 64.7% gain argues against it in commonsense classification. In our setting the explanation also serves as the retrieval signal, so dropping it collapses to the single-shot baseline. We still report it, to separate the label-quality effect from the fetching effect.
Proposed Experiments
We use MuSiQue, HotpotQA, and 2WikiMultiHopQA, the benchmarks shared by IRCoT and L-RAG. Each has two settings: the standard distractor pools, and an enlarged open pool of passages whose stored KV exceeds a fixed GPU budget. HotpotQA yes/no and comparison questions provide the classification slice, scored by accuracy. All other questions are scored by exact match and F1. We additionally report hop-2 supporting-passage recall among promoted blocks, GPU memory, bytes moved over PCIe, and per-query latency.
We will test on at least two backbone sizes in the 7–14B range.
Decisive check 1: is the premise real? Rationale-driven promotion must raise hop-2 recall well above question-driven promotion. RepairKV's stale-query result predicts that the question-only condition fails, and our premise predicts that the rationale condition succeeds. If the gap is small, the proposal is wrong.
Decisive check 2: what does splicing cost? We compare F1 against IRCoT-style re-prefill of exactly the passages we promoted. This isolates the cost of the precomputed-KV approximation from the cost of selection.
Decisive check 3: does the early trigger pay off? We test whether the middle-layer trigger lowers latency without lowering recall, across thresholds, plotting latency against F1.
Secondary analysis: protecting rationale tokens. Following the R-KV versus DefensiveKV tension, we compress the rationale tokens themselves under each policy at 10–40% budgets. We measure whether the final label degrades, which tests whether CAGE-style robustness of explanation-conditioned labels survives aggressive eviction.
References
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Cache You Later: Post-Compression KV Repair for Long-Context Agentic LLM Inference
- InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation
- Explain Yourself! Leveraging Language Models for Commonsense Reasoning
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
- DefensiveKV: Taming the Fragility of KV Cache Eviction in LLM Inference