Many questions cannot be answered from one retrieved document. "In what city was the director of Inception born?" needs a first search that finds the director and a second search that finds where that person was born. The second search cannot be written until the first one has succeeded, because the name Christopher Nolan does not appear in the question. This is the multi-hop reasoning gap. Current systems close it in one of two ways. Some let a language model write a free-text sub-query at every hop, which works but is slow and lets a wrong intermediate guess spread silently into later hops. Others update the query directly as a vector, which is very fast but says nothing about what has been found or what is still missing.
We propose a middle path, borrowed from explainable recommender systems. There, an item is described as a few named attributes plus one learned leftover vector that captures whatever the names miss. We apply the same split to the search state of a multi-hop question. The query at each hop has two parts: an explicit chain of slots (for example Inception.director = ? followed by ?.birthplace = ?) and a small learned residual vector for meaning that does not fit a named relation. A slot counts as filled only when a cheap entailment check confirms it against a retrieved passage. Retrieval stops when every slot in the chain is verified. The verified chain is the explanation, and on 2WikiMultiHopQA it can be scored directly against gold Wikidata triples.
Core Claim
A multi-hop retriever whose state is "explicit verified slots plus one implicit residual" will do three things. It will match LLM-rewriting systems on answer accuracy at a small fraction of their per-hop cost. It will beat pure vector updating on questions that need synthesis rather than entity lookup. And it will produce intermediate explanations that are both checkable against ground truth and causally used by the final answer.
The two parts of the state guard against different failures. The explicit slots, with their verification step, stop a wrong intermediate entity from passing silently to the next hop. The residual exists for hops whose target cannot be named as a single relation. Each part can be removed independently, so the claim can be falsified piece by piece.
How This Relates to Prior Work
The decomposition idea comes from Disentangled Item Representation for Recommender Systems. That paper represents each item as $N$ explicit metadata attributes plus one implicit attribute and reports fewer parameters and better cold-start accuracy. We reuse the "named attributes plus one residual" structure, but apply it to a changing search state rather than a fixed item embedding. We also drop its costly discrete reallocation step, because our residual is produced on the fly by a small network. PreferThinker motivates the "predict a structured profile, then assess each dimension" pattern. It also illustrates the known weakness of fixed attribute vocabularies, which in its case are five visual dimensions. Our slot relations are therefore open strings proposed by a planner, not a fixed ontology, and the residual absorbs what the strings miss.
The closest multi-hop systems each have one ingredient but not the combination. REAP has a fact extractor label each hop DirectAnswer, PartialClue, or Failed, and re-plans on failure. That status is a coarse label over a free-text fact. Ours is a per-slot binding that can be scored against a gold triple. Think Straight, Stop Smart uses fixed templates and stops when successive sub-query embeddings exceed cosine similarity 0.85. We instead stop when the slot checklist is verified, which is a semantic stopping rule rather than a "the query stopped changing" rule. TreeHop updates the query entirely in embedding space, subtracting the retrieved chunk's vector and adding a gated term. It cuts latency by about 99% but is uninterpretable and weaker on MuSiQue. Our residual is a TreeHop-style update confined to the leftover channel, while the subtraction of satisfied content happens symbolically by marking slots done. KBQA-R1 shows that relation-level actions with confidence gating that rejects hallucinated relations work well, but only over Freebase schemas. We carry the gating idea to unstructured text, where relations are verified against passages rather than looked up in a schema.
IRCoT and ITER-RETGEN motivate the work. They show that iterative retrieval closes much of the gap, with IRCoT gaining 11–21 recall points. They also show the failure we target: intermediate text is treated as an opaque string, so hallucinated content is fed back into the next query. STEP finds interpretable bridge phrases with a Steiner tree, but runs once before retrieval and cannot recover if the bridge is missing from the initial pool. Our loop re-retrieves per slot.
Research Program
The system has four components. The first is a planner, a language model called once per question. It writes the slot chain: an ordered list of (subject, relation, object) slots in which later subjects refer back to earlier objects. It may also emit an "unnamed constraint" phrase for parts of the question that do not fit a relation, such as "the earlier of the two" or "the one that was later renamed."
The second is the hop loop, which uses no generative model. At hop $t$ the retrieval query is
$$q_t = E\big(\mathrm{text}(s_t)\big) + r_t ,$$
where $E$ is the shared dense passage encoder. $\mathrm{text}(s_t)$ is the current open slot written out with its already-bound subject, for example "Christopher Nolan birthplace". $r_t$ is a residual vector produced by a small gated cross-attention network, in the style of TreeHop, from the residual phrase and previously retrieved passages.
The third component pairs an extractive reader that proposes a value for the open slot from the top passages with an entailment verifier that checks whether the passage supports the filled triple. A slot binds only when the entailment score exceeds a threshold $\delta$. On failure, the loop re-retrieves for that same slot using the next candidates, without advancing. After a fixed budget of failures it returns control to the planner once to re-plan.
The fourth component is a final reader. Retrieval terminates when every slot is bound, and the final reader, again one language model call, answers from the bound passages only.
Training is lightweight. The residual network is trained contrastively on gold supporting paragraphs from the HotpotQA, 2WikiMultiHopQA and MuSiQue training sets, with the slot text held fixed so that the residual learns only what the slot text misses. The planner is prompted, and optionally fine-tuned on 2WikiMultiHopQA's gold triples. The verifier is an off-the-shelf entailment model calibrated on held-out 2WikiMultiHopQA evidence.
The main analysis asks where the residual's contribution concentrates. The prediction is that it matters on MuSiQue and on the Comparison and Temporal queries of MultiHop-RAG, and is near zero on simple bridge questions. If the residual helps uniformly, that would suggest it is just extra retrieval capacity rather than a channel for unnamed constraints.
Baselines
The naive reference point is single-shot dense retrieval with a cross-encoder reranker over the same index, followed by one reader call. On MultiHop-RAG, the benchmark paper reports Hits@10 of 0.7467 and Hits@4 of 0.6625 with reranking. We will reproduce this with our encoder so that every system shares the same retrieval floor.
The strongest accuracy competitor is REAP, because it is the closest structured, re-planning design and reports the highest numbers in the file. Those figures are F1 68.0% on HotpotQA, 79.6% on 2WikiMultihopQA and 38.3% out-of-domain on MuSiQue, all from REAP. Beating it means matching or exceeding its F1 on 2WikiMultiHopQA and MuSiQue while using at least three times fewer language model calls per question.
The strongest efficiency competitor among text-based systems is TSSS, which reports 34.1 EM on HotpotQA and 9.5 s per MuSiQue sample, against 27.2 s for Iter-RetGen. For TreeHop, the file reports about 0.02 s per query against about 5 s for Iter-RetGen, and "superior recall" on 2WikiMultiHop and MultiHop-RAG. However, no reported numeric recall result was found on Lacuna for TreeHop, so we will reproduce it with its released configuration.
We will also run IRCoT and ITER-RETGEN at $T=2$, since that paper reports convergence by the second iteration. All systems share the same corpus and passage index, the same encoder where the method allows it, and the same backbone language model for any generative step. Each gets an equal prompt- and threshold-tuning budget on one shared dev split, and latency is measured on the same hardware.
Three ablations isolate the ingredients. The first removes the residual, leaving slots only; this is the "symbolic subtraction" arm. The second removes the slots, keeping the residual with the raw question, which is effectively TreeHop with our encoder. The third keeps slots and residual but removes verification, so a slot binds to the reader's top span unconditionally. A fourth variant replaces the extractive reader with a language model call per hop, to measure what the cheap loop costs in accuracy.
Why not just…?
Why not just use a bigger language model with Self-Ask or IRCoT-style prompting? Evidence in the file says scale alone does not close the gap. IRCoT's 3B Flan-T5 with iterative retrieval beat 175B GPT-3 with one-step retrieval. On MultiHop-RAG, GPT-4 reached only 0.56 with retrieved chunks, against 0.89 with gold evidence, which locates the bottleneck in retrieval rather than reasoning. TSSS reports Self-Ask at 25.6 EM on HotpotQA against 34.1 for structured reasoning. We still include a large-backbone IRCoT run. If it beats us on accuracy, the remaining case for our system rests on cost and checkable intermediates, and we will report it that way.
Why not just have the language model write a JSON list of sub-questions and run them in order? This is the most serious threat. It is close to the "no residual, language model per hop" corner of our ablations, so it is run directly. If that arm matches the full system on MuSiQue and MultiHop-RAG, then the residual is unnecessary. The contribution would shrink to two parts: verification as the stopping rule and as a guard against cascading errors, and executing the hops without generation. We would report that outcome. What distinguishes our design even then is the verification gate. The decomposition baseline advances on whatever the hop returns, and we will measure how often its final answer rests on an intermediate binding that is wrong according to the gold 2WikiMultiHopQA triples.
Why not just use TreeHop, which is already fast? TreeHop's authors report weaker results on MuSiQue, and its intermediate state cannot be inspected. Our slot-free ablation is TreeHop under matched conditions. Our claim requires the full system to beat it on MuSiQue while giving up only a modest fraction of its speed. If the gap on MuSiQue is small, the explicit slots are paying for interpretability alone.
Why not just retrieve a larger pool and rerank? The MultiHop-RAG ceiling of 0.7467 Hits@10 and HotpotQA's full-wiki Joint F1 of 17.73 show that single-pass retrieval misses bridge documents that are absent from the question's neighborhood. The naive baseline quantifies this on our index.
Proposed Experiments and Decisive Checks
We evaluate answer EM and F1 and supporting-passage recall on HotpotQA, 2WikiMultiHopQA and MuSiQue in the open-corpus setting. We also evaluate on MultiHop-RAG broken down by query type, and use Bamboogle as a small held-out probe. Cost is reported as language model calls, tokens and wall-clock time per question. Out-of-domain transfer is tested by training the residual network on HotpotQA and 2WikiMultiHopQA only and testing on MuSiQue, which mirrors REAP's out-of-domain setting.
Explanation quality is measured in two ways. Slot-binding accuracy is scored against 2WikiMultiHopQA's gold (subject, property, object) triples; the dataset paper reports that baseline evidence-path prediction is "very low" but gives no number. A causal-use test replaces a verified intermediate binding with a plausible wrong entity and checks whether the final answer changes accordingly. A high flip rate means the explanation is what the answer depends on, rather than a story told after the fact.
Three outcomes would decide the proposal. First, on questions where an intermediate hop's top passage is a distractor, does the verification gate reduce final-answer errors relative to the no-verification ablation and to IRCoT? This is the cascading-hallucination claim. Second, does the full system beat the slot-free arm on MuSiQue and the residual-free arm on MultiHop-RAG Comparison and Temporal queries? This is the claim that both channels are needed. Third, does it reach REAP-level F1 with one planning call and one answering call? This is the cost claim. Failing the first would sink the main motivation. Failing the second would reduce the method to one of its halves, and we would say which.
References
- Disentangled Item Representation for Recommender Systems
- PreferThinker: Reasoning-based Personalized Image Preference Assessment
- REAP: Enhancing RAG with Recursive Evaluation and Adaptive Planning for Multi-Hop Question Answering
- Think Straight, Stop Smart: Structured Reasoning for Efficient Multi-Hop RAG
- TreeHop: Generate and Filter Next Query Embeddings Efficiently for Multi-hop Question Answering
- KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- A STEP towards Interpretable Multi-Hop Reasoning: Bridge Phrase Identification and Query Expansion
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps