Copy-Only Sub-Questions for Multi-Hop Retrieval

Many real questions need two or more lookups. To answer "In what city was the director of Inception born?", a system first has to find the director and only then search for that person's birthplace. The standard fix is iterative retrieval: a large language model writes a free-text intermediate question or reasoning sentence, and that text becomes the next search query. It works, but it is slow, and the model can invent entities that appear in neither the question nor any document. Such an invented entity then steers every later search.

We propose a stricter alternative. Every intermediate search query may contain only two kinds of words: words copied from the original question, and a short answer span copied out of a sentence already retrieved. A small span-extraction model builds the question's decomposition tree, a keyword search engine and an extractive reader carry out each node, and a sparse linear classifier on a handful of surface features decides whether to accept a hop, retry it, or stop. Because the queries contain no generated text, an intermediate query cannot invent an entity. That makes the method cheap, and each hop can be audited. The empirical question is how much of the accuracy of generative iterative retrieval survives this constraint, and at what fraction of the cost.

Core Claim

We claim that most of the benefit of iterative decomposition on connected multi-hop questions comes from getting the bridge entity into the next query. A fluent, freely generated sub-question is not needed for that. If this holds, a copy-only query language executed over sentence-level units can recover most of the retrieval-recall gain of generative interleaving. It should also make zero intermediate-entity hallucinations and use orders of magnitude fewer tokens. The claim is falsifiable. If copy-only queries lose most of the gain on MuSiQue, a benchmark built to block single-hop shortcuts, then generation is doing real work that copying cannot replace.

The three input directions play different roles. Hierarchical query decomposition supplies the mechanism, a tree of sub-questions with slots that are filled bottom-up. The multi-hop reasoning gap supplies the regime and the failure we target: the next-hop entity is absent from the original query. Minimalist feature engineering, the line of work showing that small hand-chosen feature sets can match large learned representations, supplies a design constraint on both the query language and the controller. We treat that constraint as a hypothesis to test, not an article of faith.

Where This Sits

Three papers motivate the gap. IRCoT alternates one generated reasoning sentence with one retrieval call. It raised retrieval recall by 11–21 points and QA F1 by up to 15 points on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC. Its queries are unconstrained sentences over full paragraphs, which is exactly the source of drift and context bloat we remove. Question Decomposition for RAG decomposes the question once, upfront, into a fixed set of sub-questions and reranks the pooled results with a cross-encoder. It reports 18.9 s per query against 0.03 s for plain retrieval. Because it never sees intermediate results, it cannot fill a bridge slot. HANRAG routes queries by type and filters passages with a relevance discriminator. All of its routing and filtering components are prompted or fine-tuned language models, and we replace them with surface features.

Two papers supply our method. DECOMPRC casts decomposition as pointing to spans of the original question, needing about 400 labeled examples. It reached 70.6 F1 in the HotpotQA distractor setting, and 55.8 F1 under adversarial question inversion where BERT fell to 19.3. It worked only over ten pre-selected paragraphs and could not fetch a bridge entity from outside them. Our contribution is to fill DECOMPRC's slots from open-corpus retrieval. Is a Question Decomposition Unit All We Need? showed that atomic sub-questions let a SQuAD-trained RoBERTa handle complex questions. The same paper reports that automatic generative decomposers suffered from incoherence and error propagation, which is why we avoid generation at the intermediate steps.

Two papers constrain the design. Context-Efficient Retrieval with Factual Decomposition and Hierarchical Lexical Graph show that small factual units beat 100-plus-token chunks. Both paid for offline LLM extraction, and both use static retrieval rather than a dynamic query tree. We therefore default to plain sentences, which cost nothing to index, and test the LLM-extracted units only as an ablation. MuSiQue raises a real warning for any keyword-matching method: single-hop models lose 30 F1 there compared with HotpotQA, so surface overlap on older benchmarks can reflect shortcuts rather than multi-hop reasoning. GRITHopper is the direct opposite of our design: dense, decomposition-free hopping with a fine-tuned 7B model. It reports 95.44 Hits@1 on MoreHopQA and MuSiQue retrieval in 6 min 42 s against 1 h 24 min for GPT-4o decomposition.

Research Program

The system has four parts.

The first is a span-pointer decomposer, trained DECOMPRC-style. It maps a question to a small tree whose nodes are question spans. Some nodes carry a slot marker meaning "the answer to child node $j$ goes here." We obtain supervision cheaply by aligning MuSiQue's single-hop components to spans of the composed question, and we add a few hundred HotpotQA bridge and comparison examples.

The second is the executor, which processes the tree leaves-first. For each node it issues a query made of the node's question tokens plus any filled slot. It runs BM25 keyword search over a sentence index, tagging each sentence with its article title. It then runs a small extractive reader to pull an answer span from the top sentences, and that span fills the parent's slot.

The third is the controller, a sparse logistic model over roughly a dozen hand-chosen features. The features include the reader's span confidence and its margin over the runner-up. They also include lexical overlap between the node span and the supporting sentence, and agreement between the question word and the answer's entity type. One feature flags whether the answer string already appears in the question, a direct signal of a shortcut. The rest cover the retrieval score gap and the number of unfilled slots.

The controller makes one of three choices per node: accept the answer, retry with the next-ranked answer span (bounded beam), or halt the tree as unanswerable. It is trained on MuSiQue-Full, where answerable questions are paired with unanswerable contrast questions, so it learns to abstain.

The fourth part handles questions whose structure is implicit, where no span decomposition fits. When the decomposer's confidence falls below a threshold, the question is passed to an LLM decomposer, and we report how often this fallback fires. Because the fallback fraction is reported, the cost savings cannot be hidden by quietly routing hard questions to the expensive path. An LLM may compose the final answer from the collected sentences, but only at the root.

The analysis side has two parts. First, we attribute errors to tree nodes: for each failed question, we identify the first node whose slot filler disagrees with gold. This is only possible because every hop is an explicit span. Second, we measure hallucination in intermediate queries, defined as the fraction of query entities that occur in neither the question nor any evidence retrieved so far. This rate is zero for our method by construction and measured for the generative baselines.

Baselines

The naive reference point is one-shot retrieval: BM25 over the same sentence index using the full question, and separately a dense retriever, both feeding the same final reader. This shows how much any iteration buys.

The strongest competitor on our specific question, whether generation is needed in the intermediate query, is IRCoT. We run it with its original design but swap in our sentence index and our final answer model, so the only difference is how queries are formed and when to stop. The file gives only IRCoT's relative gains (11–21 recall points, up to 15 F1). We therefore re-run it ourselves and report absolute numbers, and we tune its prompt and its maximum number of steps with the same development budget as our controller. Beating IRCoT means matching its supporting-fact recall on MuSiQue within two points while using at least ten times fewer tokens and less wall-clock time. Exceeding its answer F1 would be a bonus, not the bar.

QD+RR is the reference for upfront decomposition. Its reported figures are 87.2% Hits@10 and 0.635 MRR@10 on MultiHop-RAG, and 28.1 EM / 35.0 F1 on HotpotQA (Question Decomposition for RAG). We compare directly on these metrics and settings.

GRITHopper is the reference for dense, decomposition-free hopping. Its reported figure is 95.44 Hits@1 on MoreHopQA against 82.16 for the Multi-hop Dense Retriever (GRITHopper). We do not expect to beat it on that metric with BM25. We compare accuracy per unit of compute and hop-level auditability, and state that trade-off plainly.

HANRAG is compared on MuSiQue using its protocol. Its gains in the file are reported relative to Adaptive-RAG (about 7% EM, over 16% accuracy), so there is no reported absolute HANRAG MuSiQue score on Lacuna, and we re-run it where code permits.

Three ablations isolate our ingredients. The key one replaces copy-only queries with free sub-questions generated by a small LLM, keeping decomposer, retriever, reader and controller fixed. The second replaces the sparse controller with a fine-tuned small transformer classifier fed the same inputs. The third swaps sentences for LLM-extracted entity–description units (Context-Efficient Retrieval with Factual Decomposition) to see whether offline extraction earns its cost.

Why not just…?

Why not just run IRCoT with a small, cheap language model? This is the main threat. The IRCoT paper already shows a 3B Flan-T5 with interleaving beating 175B GPT-3 with one-shot retrieval, so the cost gap may be smaller than we assume. The generated-query ablation answers it directly. Under an identical pipeline, if small-model generated queries beat copy-only queries on MuSiQue recall by more than a few points, then generation is doing real work. We would then report copy-only as a cost floor rather than a replacement. The hallucination-rate measurement shows what that generation costs in reliability.

Why not just use decomposition-free dense hopping like GRITHopper? On raw retrieval accuracy it may simply win; its MoreHopQA Hits@1 of 95.44 is high. Its costs are fine-tuning a 7B model and maintaining a dense index, and its hops are not inspectable. Our case rests on the accuracy-per-compute curve and on node-level error attribution, which GRITHopper cannot provide. If GRITHopper is both cheaper at inference on our hardware and more accurate, the minimalism argument fails for retrieval-only use. It would then survive only where auditing matters.

Why not just use a fine-tuned small transformer as the controller instead of hand-picked features? This might work equally well, and if so we will say so. The minimalist literature, such as the question-difficulty study, where handcrafted features beat BERT embeddings, gives a reason to expect parity, but that evidence comes from classification, not from routing retrieval hops. For this reason we do not stake the core claim on the controller. The controller-capacity ablation decides whether the sparse controller is merely adequate or actually better out of domain, measured by training on MuSiQue and testing on 2WikiMultihopQA and MultiHop-RAG.

Why not just retrieve more and rerank with a cross-encoder? MultiHop-RAG reports that even with a strong reranker, hit rates across multiple evidence pieces stay poor. It also reports that GPT-4 drops from 89% accuracy on gold evidence to 56% on retrieved evidence. A reranker cannot surface a bridge document that no query ever pointed at. The one-shot-plus-reranker configuration is nonetheless included in the naive baseline tier.

Proposed Experiments

The main benchmarks are HotpotQA in the full-Wikipedia setting (HotpotQA), 2WikiMultihopQA, and MuSiQue-Ans and MuSiQue-Full. MultiHop-RAG serves as an out-of-domain news corpus whose comparison and temporal questions stress span decomposition, and MoreHopQA tests generalization to longer hop chains. Metrics are supporting-fact recall and support F1, answer EM and F1, abstention accuracy on MuSiQue-Full, tokens and wall-clock time per question, fallback rate, and intermediate hallucination rate.

There are three decisive checks. First, the copy-only versus generated-query ablation on MuSiQue tests the core claim on the benchmark least friendly to surface matching. Second, a shortcut probe follows up MuSiQue's disconnected-reasoning concern. It compares accuracy on connected versus disconnected-solvable subsets and tracks how often the "answer already in question" feature drives acceptance, which shows whether minimal lexical features reward shortcuts or suppress them. Third, a hop-length curve on MoreHopQA tests whether copy-only slot filling degrades gracefully as the tree deepens, since every hop's error compounds and nothing regenerates a bad query. A result where copy-only matches IRCoT on HotpotQA but collapses on MuSiQue would be informative in its own right. It would indicate that the old benchmark's shortcuts, not the method, explained the parity.