Speculative Multi-Hop Retrieval: Drafting Whole Query Trajectories with a Small Model

Iterative retrieval-augmented generation answers hard questions in rounds. A language model writes a search query, reads the passages a retriever returns, writes a short intermediate answer, writes the next query, and repeats until it can answer. This loop is much more accurate than searching once, but it is slow. Every round forces the large model to decode new text one token at a time and then wait for retrieval before it can continue. We propose to move most of that sequential work onto a small model. The small model runs the entire multi-hop loop by itself, producing every query, every retrieval call, and every intermediate answer. The large model then checks the whole drafted trajectory in one parallel scoring pass, without generating anything. If the large model would have written the same queries, the drafted hops are accepted for free. At the first hop where it would not, the large model takes over from that point.

This is the logic of speculative decoding (a small model guesses tokens and a large model verifies them in parallel), lifted from the level of tokens to the level of retrieval hops. Retrieval is deterministic given a query, so accepting a hop only when the large model's greedy output matches the drafted query exactly makes the final answer identical to what the large model's own slow loop would have produced. The speedup is bought without any change in accuracy. The question this note asks is empirical and sharp: how often does a small model write the same multi-hop search trajectory as a large one, and does verifying whole hops beat verifying tokens?

Core Claim

The latency of iterative query refinement comes from two sequential costs. The first is that the large model decodes queries and sub-answers autoregressively. The second is that the large model makes one call per hop, with a retrieval round trip in between. Both can be cut by trajectory-level speculation, and with exact-match acceptance this does not change the output at all. Concretely, let $T$ be the number of hops in the large model's own trajectory and $a$ the number of leading hops the small model drafts identically. The large model then needs one verification pass plus a fresh continuation from hop $a+1$, instead of $T$ decode-heavy calls. Speedup should therefore grow with the per-hop agreement rate between the two models.

This differs from the closest work in specific ways. Speculative RAG uses a small drafter that writes answer drafts in parallel over clusters of passages from a single retrieval round. A large verifier then picks one, cutting latency by up to 50.83% on PubHealth. It never re-queries, and its verifier selects among drafts rather than guaranteeing agreement with the large model. Stop-RAG learns when to stop iterating, using an offline value estimate. It leaves each hop's cost untouched and treats query formulation as a black box. Our method is orthogonal to it and can be stacked with it. Meta-Reflection removes multi-pass reflection by storing reflective hints inside the model's hidden states. It never touches an external retriever. SpecAgent removes retrieval latency by precomputing context at indexing time. That works for code repositories, but not for open-domain questions that are unknown until they arrive.

Research Program

The base system is a standard iterative loop: generate query, retrieve top-$k$, generate sub-answer, and either continue or answer. This is the family of closed-loop refiners motivated by CoV-RAG and IM-RAG. We pair a small drafter with a large target that share a tokenizer, so that exact token agreement is well defined. The Mistral-7B / Mixtral-8x7B pairing used by Speculative RAG is the natural choice.

Drafting runs the full loop with the small model, including real retrieval calls. Verification concatenates the question, drafted queries, retrieved passages, and drafted sub-answers into one sequence. The large model scores this sequence in a single prefill pass. Prefill processes all tokens in parallel, so it is far cheaper per token than decoding. The pass finds the first model-written token where the large model's greedy choice differs from the draft. Everything before that token is accepted. The large model's own token is inserted at the divergence point, and its decoding resumes from there with its existing cache. Passages are only retrieved again if the divergence changed a query.

We will build three variants on this core. The exact variant is lossless by construction and is the main object of study. The relaxed variant accepts a drafted query when the large model assigns it high enough per-token probability, even if it is not the greedy choice. This trades exact fidelity for a higher acceptance rate, and we will report the accuracy-latency curve this produces. The branching variant has the drafter sample several candidate next queries at each hop and retrieve for all of them in parallel. The large model verifies every branch in one batched pass. This mirrors the multi-perspective sampling of Speculative RAG, but across hops rather than within one round, and is meant to raise acceptance at the cost of extra parallel compute.

The analysis component asks where drafter and target disagree: in entity choice, in query phrasing, in sub-answers, or in the decision to stop. It also asks whether disagreement concentrates at particular hop depths. This tells us whether a small amount of distillation of the drafter on target trajectories would raise acceptance. Unlike the reinforcement-learning training in IM-RAG or SeRTS, such distillation needs no annotated gold support passages.

Baselines

The naive reference point is the large model running the iterative loop greedily by itself, with no speculation. Its accuracy is the target the exact variant must reproduce identically. Its wall-clock latency is the denominator for every speedup we report.

The strongest competitor on the latency question is token-level speculative decoding applied inside each hop. It uses the same small drafter and the same large target, so it is equally lossless. If hop-level speculation is no faster than this well-understood technique, the proposal fails. We will run both on identical hardware with the same draft model, the same retriever and index, and a draft length tuned on the same held-out split. Beating it means lower median and tail latency per question at identical output.

The strongest accuracy-oriented competitor in the file is IM-RAG. It reports F1 of 82.5 and Exact Match of 68.4 on HotpotQA, after training a questioner with reinforcement learning, but it reports no latency. On MuSiQue, Stop-RAG reports F1 47.0 against 44.8 for prompted stopping. We will run Stop-RAG's stopping controller on top of both the plain loop and our speculative loop. This tests whether the savings add up. We do not claim to beat IM-RAG's accuracy; our claim is equal accuracy to the chosen target at lower latency.

Speculative RAG is the closest latency method. It reports 50.83% lower end-to-end latency and 12.97% higher accuracy than standard RAG on PubHealth. We will run it with the same model pair. This shows the accuracy cost of being single-round on multi-hop data, and its speed relative to our loop.

The key ablation is the small model running the loop alone, with no verification. It isolates what verification buys in accuracy and sets the latency floor.

Why not just…?

Why not just use token-level speculative decoding inside each hop? This is the real threat, and we treat it as the main comparison rather than arguing around it. Token-level speculation still needs at least one large-model call per hop, separated by retrieval waits. When drafts are rejected, it needs several calls per hop. Hop-level speculation can, in the best case, compress a whole trajectory into one prefill. The cost is that a rejection at hop $j$ throws away the drafter's work on all later hops. Which effect dominates depends on the per-hop acceptance rate, which nobody has measured. If token-level speculation wins at realistic acceptance rates, we will report that result.

Why not just let the small model run the whole loop? It would be fastest. But the evidence in the file suggests small models lag on multi-hop retrieval. SeRTS describes a retrieval performance gap between small and large models. IM-RAG reports only 37.1 F1 for prompted GPT-3.5 on HotpotQA, versus 82.5 for its trained system. The small-model-only ablation will quantify this gap on our pair. If the gap is negligible, the large model is unnecessary and the proposal is moot. That is a useful finding in its own right.

Why not just issue several rewritten queries at once, in a single round? This removes the sequential dependency entirely. However, CoV-RAG found that rewriting before retrieval underperforms revising after seeing a draft. Chain-of-Rewrite found that a second, evidence-informed pass raised PopQA Recall@5 from 37.4% to 59.8%. Later hops of a multi-hop question depend on entities that only appear in earlier evidence. We will still include single-round multi-query fan-out as a baseline on MuSiQue and 2WikiMultihopQA, where this dependency should hurt most.

Why not just stop earlier? Stopping controllers such as Stop-RAG reduce the number of hops. Stop-RAG itself slightly underperformed on HotpotQA, where continuing to retrieve paid off. Stopping early trades accuracy for speed. Our method makes each hop cheaper without changing which hops happen, so the two compose, and the stacked experiment tests that claim.

Proposed Experiments

We will evaluate on HotpotQA, MuSiQue, and 2WikiMultihopQA as the multi-hop core, and on PopQA and Natural Questions as shallower controls where few hops are needed. A shared dense index and fixed top-$k$ are used throughout. Metrics are answer F1 and Exact Match, plus wall-clock latency per question (median and 95th percentile) measured end to end, including retrieval. We also count large-model forward passes, large-model decoded tokens, and per-hop acceptance rate.

The first decisive check is fidelity: the exact variant must produce byte-identical answers to the greedy large-model loop on every question. Any deviation indicates an implementation error, not a tradeoff.

The second decisive check is the speedup curve against hop count, compared directly with token-level speculation. Hop-level speculation should pull ahead as trajectories lengthen on MuSiQue, and should offer little on PopQA. If it does not pull ahead on long trajectories, the central claim is false.

The third check is the acceptance analysis. It reports where drafter and target diverge, and how much distilling the drafter on target trajectories raises acceptance and speedup.

The fourth check plots the relaxed and branching variants as accuracy-latency curves against Speculative RAG and the stacked Stop-RAG configuration. This shows whether giving up exactness buys meaningfully more speed than the lossless version.