A Recurrent Hop State for Multi-Hop Retrieval Without Context Accumulation

Questions like "In what city was the director of Inception born?" cannot be answered by one search, because the second document (Nolan's birthplace) only becomes findable after the first hop reveals the bridge entity. Current systems handle this in one of two ways. Some write a new text query at every hop and paste every retrieved passage back into the prompt, which is slow and makes the prompt grow with each hop. Others read the next search target straight out of a language model's hidden activations, which is fast but has only been shown for a single extra hop. We propose a third design. A small recurrent cell carries a compact "hop state" from one hop to the next and turns it directly into a search vector for an external document index. At each hop the language model sees only the question and the current passage, never the growing pile of earlier ones.

The bet is a division of labor that the theory in this file points to. Recurrent networks track sequential state well but are poor at storing large volumes of facts. External indexes store facts well but have no notion of where a reasoning chain currently stands. Long prompts are where attention-based models make sporadic, hard-to-remove errors. If the recurrent cell keeps track of the chain and the index holds the facts, accuracy should fall off more slowly as hop count and distractor load grow, and each hop should cost about as much as one search.

Core Claim

Our claim is that a learned recurrent hop state, updated from the language model's intermediate-layer reading of each retrieved passage and used directly as a dense retrieval query, will chain 3–6 hops with less degradation per hop than either text-query loops that accumulate context or memoryless latent retrieval. It will also do so at roughly one forward pass per hop. The specific new element is that the state persists and is updated across hops while raw passages are dropped. The closest prior systems each have only one half of this. One reads latent queries but does not persist them. The other persists memory but only over a small pre-filtered candidate pool.

Three bodies of work motivate the split. Separations in the Representational Capabilities of Transformers and Recurrent Architectures proves two things. A recurrent model tracks bounded nesting with a constant-size state. The same recurrent model needs state that grows linearly with length to perform lookup, and Mamba's lookup accuracy collapses past length 100. This result constrains our design: the recurrent cell must never be asked to store facts, only to record where the chain stands. Exposing Attention Glitches with Flip-Flop Language Modeling shows that Transformers holding one bit across filler tokens make a persistent tail of errors. A 1-layer LSTM makes none, and attention-sharpening regularizers reduce the errors without removing them. This is the motivating evidence that accumulated-passage prompts are a liability. Generalization through Memorization: Nearest Neighbor Language Models and Generalization Properties of Retrieval-based Models argue that fact storage belongs in an external key-value store rather than in weights. We adopt that position for the document index.

Relation to the Closest Work

Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) finds that middle Transformer layers already encode the next-hop entity. It maps those activations through an MLP into the embedding space of a dense retriever (Contriever), skipping text generation. We reuse this mechanism as our per-hop "read." L-RAG is a single static probe evaluated mostly on 2-hop questions, and the retrieved document is simply appended for another forward pass. We add state that persists across hops, a learned halting decision, and removal of past passages from the prompt.

Memory Augmented Sequential Paragraph Retrieval for Multi-hop Question Answering (GMF) keeps a rewritable vector memory with a threshold write gate. It reports 86.3% paragraph exact match on HotpotQA full-wiki, and removing the rewritable memory costs 6.6% precision. That supports the value of a persistent vector state. However, GMF scans a candidate set already filtered by TF-IDF, so a bridge document with no lexical overlap is never seen. Our state instead issues queries against the full dense index.

Memory-Aware and Uncertainty-Guided Retrieval for Multi-Hop Question Answering (MIND) stores verified bridge facts as text in the prompt and triggers retrieval from model uncertainty. Our approach moves that memory out of prompt space.

Universal Transformers supplies the idea of learned halting (Adaptive Computation Time) over recurrent steps, but it has no external memory. Our halting signal is tied to whether the retrieved evidence resolves the chain.

Research Program

The mechanism has three parts. Let $q$ be the question and $d_t$ the passage retrieved at hop $t$. Let $m_t$ be the mean-pooled middle-layer activation of a frozen-backbone language model reading the pair $(q, d_t)$. Let $s_t$ be a hop state of a few hundred dimensions. The update and the query are

$$s_t = \mathrm{GRU}(s_{t-1}, g_t \odot m_t), \qquad k_{t+1} = W s_t,$$

where $g_t \in [0,1]$ is a learned write gate in the spirit of GMF, $W$ projects the state into the frozen retriever's embedding space, and $k_{t+1}$ is the query vector used for nearest-neighbor search over the passage index. A halting head on $s_t$ outputs the probability that the chain is resolved. When it fires, the reader generates the answer from $q$, the final passage, and a soft prompt derived from $s_t$. Earlier passages are never put back in the prompt.

Training uses per-hop supervision. MuSiQue is built by composing single-hop questions (♫ MuSiQue: Multihop Questions via Single-hop Question Composition), and 2WikiMultihopQA ships Wikidata evidence triples (Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps), so both give the gold paragraph for each hop. We train with three losses. A contrastive loss pulls $k_{t+1}$ toward the gold next-hop paragraph and away from hard negatives. A halting loss uses the true hop count. An answer loss trains the reader with LoRA adapters on the backbone. Training starts with teacher forcing on gold passages and then moves to scheduled sampling on the model's own retrievals, so the state learns to recover from a wrong hop rather than only to follow correct ones. The backbone is Llama-3.1-8B-Instruct, chosen to match Chain-of-Retrieval Augmented Generation (CoRAG).

MuSiQue tops out at 4 hops. To test where the recurrent bias should matter, we also build a longer-chain evaluation by composing 2WikiMultihopQA's Wikidata relations into 5- and 6-hop chains over the same Wikipedia index. All held-out relation compositions are unseen in training. This is the natural-language analogue of a flip-flop test: the chain must stay on track while each hop brings in a fresh, possibly misleading passage.

Baselines

The naive reference points are no retrieval and single-step dense retrieval with the same reader. CoRAG reports single-step retrieval recall of 29.0% on MuSiQue, and our setup should reproduce something close to that, which serves as a check that the pipeline is calibrated.

The strongest existing method is CoRAG-8B. It is fine-tuned on rejection-sampled chains to write sub-queries and sub-answers, and its accuracy grows with inference-time search. In the file it reports 47.1% retrieval recall on MuSiQue via sub-query decomposition and an advantage of over 10 exact-match (EM) points against baselines on MuSiQue and 2WikiMultihopQA. No absolute EM value for CoRAG is reported on Lacuna in this file, so we will rerun it. We run CoRAG with greedy decoding and with Best-of-$N$, on the same backbone, Wikipedia index, and training questions. Our method gets the same number of fine-tuning examples and a matched hyperparameter search. Beating CoRAG means one of two outcomes. Either we match its greedy EM on MuSiQue at a fraction of the wall-clock time and generated tokens, or we exceed it on the 5–6 hop split at equal latency. We do not expect to beat Best-of-$N$ CoRAG at its full compute budget, and we will plot accuracy against compute to show where the curves cross.

Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (IRCoT) is the prompt-based text-loop baseline. It reports retrieval recall gains of 11–21 points over one-step retrieval on HotpotQA. We run it with the same backbone and retriever.

L-RAG is the latent baseline. It reports 0.352 accuracy on MuSiQue and about 2.8 seconds per query on 2WikiMultihopQA, against more than 20 seconds for IterRetGen. We reimplement it on our backbone and also extend it naively to multiple hops by repeating the probe with accumulated passages.

Three ablations isolate the key ingredient. The first is a memoryless variant in which $k_{t+1}$ is computed from $m_t$ alone. The second keeps recurrence but also accumulates passages in the prompt. The third replaces the GRU with a small Transformer attending over all past hop vectors. The recurrence claim survives only if the full model beats the first two ablations and at least matches the third on the long-chain split.

Why not just…?

Why not just iterate L-RAG, re-probing the middle layer after appending each new passage? This is the most direct competitor, and it is included as a baseline. Its prompt grows with every hop, which is exactly the regime where the flip-flop study finds a persistent tail of attention errors. Its query at hop $t$ also depends on the model re-deriving the whole chain from raw text each time. If iterated L-RAG matches our method on the 5–6 hop split, the persistent state adds nothing and we will report that.

Why not just use CoRAG or IRCoT, since CoRAG already compresses each hop into a short text sub-answer? This is a real threat. CoRAG's sub-answers are a compressed text state, so the honest comparison is between a text state and a vector state. Two experiments settle it. The first measures accuracy at matched latency and generated tokens; L-RAG's numbers above suggest text loops cost roughly an order of magnitude more time. The second measures recovery rate, defined as accuracy at hop $t+1$ given a wrong retrieval at hop $t$. Text sub-answers commit to a possibly hallucinated string, while a gated vector state can partially discount a bad passage. If CoRAG wins at matched compute, the proposal is reduced to a speed result.

Why not just put all candidate passages into a long-context model or use Fusion-in-Decoder? Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering reads 100+ passages at once, but they all come from a single search, and the bridge document is often not among them. MuSiQue's construction removes shortcuts and causes a 30-point F1 drop for single-hop models. We include FiD over the top-100 passages from a single query as a reference point to measure how much of the gap remains retrievable at all.

Why not just attend over past hop vectors instead of using recurrence? With six or fewer hops, attention over a handful of vectors does not dilute much, so it may match the GRU. This is the third ablation. We expect it to be the closest of the three, and a tie would mean the gain comes from a persistent compact state rather than from recurrence specifically. That is still useful, and we would describe the contribution in those terms.

Decisive Checks

Three results decide the proposal. The first is the accuracy-versus-hop-depth curve from 2 to 6 hops. Our method should keep a noticeably flatter slope than IRCoT and iterated L-RAG, measured as EM and per-hop retrieval recall. The second is a distractor-injection test. We insert lexically similar but wrong passages at a chosen hop and measure how often the chain drifts off course afterward. The prediction is that methods which accumulate passages in the prompt drift more as distractors accumulate, while the gated state does not. The third is a cost accounting of seconds per query and generated tokens on 2WikiMultihopQA, which should land near L-RAG's reported ~2.8 seconds rather than IterRetGen's >20.

If the depth curve is not flatter than iterated L-RAG's, the core claim is false. If it is flatter but the attention-over-hops ablation matches it, the claim narrows to "persistent compact state" and no longer supports recurrence in particular.