A multi-hop question is one whose answer requires chaining facts: find the director of a film, then that director's birthplace. Systems that answer such questions alternate between reasoning and searching a document collection, a setup called retrieval-augmented generation (RAG). In these systems one wrong intermediate answer usually ruins every later step. The wrong entity becomes the next search query, and the passages it retrieves look plausible. The model never notices.
This proposal treats that cascade as something to measure and then train away. We first build a question generator over a fictional world, so no question can have been memorized, and measure how often an injected error at hop $k$ is later corrected. We then train with a hop-depth curriculum. Training starts on short, clean chains and gradually moves to longer chains with deliberately corrupted early hops. The hypothesis is that the curriculum produces an agent whose accuracy levels off as chains get deeper, instead of decaying geometrically. It should also generalize to chain depths never seen in training.
Core Claim
A simple model makes the target concrete. Suppose that at each hop the agent introduces an error with probability $\varepsilon$, and that, once wrong, it returns to a correct chain with probability $\rho$ per hop. Treat the chain as a two-state Markov process. With no recovery ($\rho = 0$), the probability of a correct answer after $K$ hops is $(1-\varepsilon)^K$, which falls toward zero. With recovery, the fraction of chains in the wrong state converges to
$$ \frac{\varepsilon}{\varepsilon + \rho}, $$
which does not depend on $K$. This is the discrete analogue of what On Error Propagation of Diffusion Models and Test-Time Iterative Error Correction show for diffusion samplers. When an iterative update cannot shrink its own errors, they compound. Making the update self-correcting (contractive) turns exponential growth into bounded growth.
Our claim has three parts:
- Current multi-hop agents have $\rho$ near zero.
- $\rho$ can be measured directly by injecting errors.
- A curriculum that grows depth while scheduling corruption raises $\rho$ more than either ingredient alone.
The decisive signature is the shape of the accuracy-versus-depth curve on depths beyond training. It should be a plateau, not a geometric decay.
How This Relates to Prior Work
Several papers motivate the gap.
- GRAT frames multi-hop retrieval as Monte Carlo tree search guided by process rewards, explicitly because one early retrieval error corrupts all later hops in linear chains.
- Chain-of-Retrieval Augmented Generation (CoRAG) fine-tunes Llama-3.1-8B on rejection-sampled sub-query chains and attributes drift to distractor passages entering working memory.
- IRCoT interleaves chain-of-thought with retrieval and appends everything it retrieves, so bad context is never removed.
All three handle error by searching more at test time or by filtering training data. None measures whether the trained policy itself can recover once it is off track. That recovery quantity is what we target.
The closest training-side work is EVO-RAG. It uses a two-stage curriculum, broad exploration followed by efficient refinement, with step-level rewards optimized by preference learning. Its curriculum schedules reward weights, not task depth or corruption. It also evaluates only on static Wikipedia benchmarks. We schedule what the agent sees rather than how it is scored, and we evaluate where memorization is impossible.
Two papers supply the curriculum method. GrowLength and Dataset Decomposition show that growing sequence length on a doubling schedule acts as an optimization warm-up and speeds convergence. We reuse the doubling schedule with hop depth in place of token length. One caveat from Dataset Decomposition constrains us: length curricula help little when targets are short. Our gains therefore have to come from extrapolation in depth, not from training speed, and we test for that specifically.
Three papers motivate the evaluation design.
- DyVal generates reasoning problems from directed acyclic graphs with controllable depth. Models strong on static benchmarks scored nearly zero on these problems. DyVal covers only closed-world symbolic tasks with no retrieval, and we extend it to an open corpus.
- MuSiQue shows that older benchmarks admit single-paragraph shortcuts: 64.8 F1 on HotpotQA without real multi-hop reasoning.
- GRADE and MultiHop-RAG use post-cutoff news to avoid contamination. Both only correlate errors with difficulty and cannot intervene on a specific hop.
Research Program
Generator. The first artifact is a fictional-world multi-hop generator. We sample a knowledge graph over invented entities: people, organizations, places, and dates, with typed relations. We render each fact as several short encyclopedia-style documents using paraphrase templates and an LLM rewriter, and mix in distractor documents about near-duplicate entities. A question is a path of length $K$ through the graph, phrased so that each bridge entity is unnamed. Because every entity is invented, no answer can be recalled from pretraining. Because every hop has a known gold entity and gold document, we can inject errors precisely. At hop $j$ we replace the agent's retrieved passage with a plausible distractor, or overwrite its intermediate answer with a sibling entity. We then observe whether later hops return to the gold path. This yields per-hop estimates of $\varepsilon$ (error rate) and $\rho$ (recovery rate), which are the core measurement of the project.
Training. We fine-tune Llama-3.1-8B-Instruct, the model CoRAG uses, on chains in the sub-query, sub-answer, final-answer format. Training trajectories are produced by rejection sampling as in CoRAG. Corrupted examples additionally show a recovery step: the agent checks the intermediate answer against what was retrieved, notices the mismatch, and issues a corrective re-query. The curriculum starts on depth-1 and depth-2 clean chains. It doubles the maximum depth to 4 and introduces corruption at a low rate, then raises the corruption rate and moves corruptions toward earlier hops. We start corruption late because a model must first be able to carry out a chain before it can learn to repair one. As in Dataset Decomposition, phase transitions are fixed in advance. As a secondary arm we test an adaptive scheduler that advances when validation recovery rate saturates, since Direction 1 flags static schedules as an open problem.
Out-of-domain tests. We test on depths 6 and 8 in the fictional world. We also test transfer to MuSiQue, which has 2–4 hops and shortcut filtering, and to GRADE and MultiHop-RAG, which are post-cutoff natural news. For each of these we report results by hop count and, for MultiHop-RAG, by query type.
Baselines
Do-nothing reference. The naive reference point is IRCoT-style prompting of the same untuned Llama-3.1-8B-Instruct, with the same retriever and the same maximum number of hops. It shows what retrieval interleaving alone achieves and gives the baseline $\rho$. We expect $\rho$ to be near zero because retrieved distractors are appended permanently.
Strongest training method. The strongest training competitor is CoRAG-style rejection-sampled fine-tuning on a uniform mixture of all depths with clean chains. On MuSiQue, CoRAG reports raising retrieval recall from 29.0% to 47.1% and outperforming IterDRAG and Search-o1-32B by more than 10 exact-match points. We retrain it on our fictional-world chains with the same base model, the same total training tokens, the same number of rejection-sampling attempts, and the same hyperparameter-search budget. That way its only difference from our method is the absence of ordering and corruption. Beating it means two things. First, a significantly higher $\rho$ at depths 6 and 8 under injection. Second, no loss on clean depth-2 to depth-4 accuracy and on MuSiQue.
Strongest test-time method. The strongest test-time competitor is GRAT's process-reward tree search. GRAT reports Llama3-8B rising from 60.6% to 66.8% accuracy on 2WikiMultiHopQA after self-training on 11,000 search trajectories. We give GRAT the same base model and match inference compute in tokens. We report our greedy policy against GRAT at equal compute, and our policy with Best-of-N at GRAT's compute. Since tree search can sidestep a bad branch without the policy ever learning to recover, we report both final accuracy and injected-error recovery.
Ablations of the key ingredient. Two ablations isolate what the curriculum adds. In the first, the same corrupted trajectories are shuffled uniformly with no depth ordering. In the second, the same depth curriculum is used with no corruption.
Other reference points. For context on natural data, MuSiQue's Select+Answer model reaches 47.3 F1 against 78.0 for humans MuSiQue. On MultiHop-RAG, GPT-4 reaches 0.56 accuracy with retrieved chunks MultiHop-RAG. There is no reported result on Lacuna for any method on our fictional-world generator, so the internal baselines above define the comparison there.
Why not just…?
Why not just train on noisy retrievals without any curriculum? This is the most serious threat. Corruption-augmented fine-tuning may be the entire effect. The shuffled-mixture ablation is designed to settle it. If shuffled corruption matches the curriculum on recovery at unseen depths 6 and 8, we will report that the ordering is unnecessary and that error injection plus recovery supervision is the contribution. The curriculum's specific prediction is narrower: better extrapolation in depth. This follows the warm-up argument in Dataset Decomposition, which reports that abrupt jumps in task complexity destabilize optimization.
Why not just use tree search or Best-of-N at test time? GRAT and CoRAG both show that search helps. However, CoRAG reports accuracy scaling only log-linearly with inference tokens. Every hop of search multiplies cost, and deep chains make that cost prohibitive. A policy with nonzero recovery should need less search for the same accuracy. The compute-matched comparison tests this. If GRAT at equal tokens wins on deep chains, search is the better tool, and our method's value reduces to cheaper greedy inference.
Why not just use a bigger model? IRCoT shows a 3B model with interleaved retrieval beating 175B GPT-3 with one-shot retrieval. DyVal reports GPT-3.5-Turbo losing 23% arithmetic accuracy from its shallowest to deepest level. Scale alone does not remove depth-driven decay. We will still measure $\rho$ for Llama-3.1-70B with prompting only. If a larger model has substantial recovery without training, that is informative in itself.
Why not just evaluate on post-cutoff news benchmarks like GRADE? These benchmarks prevent memorization, but they lack gold intermediate states that can be swapped out, so recovery cannot be measured directly. We use them as transfer tests, not as the instrument.
Decisive Checks
Three results determine whether the claim holds.
- Shape of the depth curve. We fit the two-state model to accuracy-versus-depth curves at depths 1–8. We expect the uniform-mixture baseline to fit $\rho \approx 0$, showing geometric decay. We expect the curriculum model to fit $\rho > 0$, showing a plateau at depths 6 and 8 that it never trained on. A geometric curve for our model would falsify the claim.
- Recovery under injection. At each injection hop $j$, the curriculum model should beat the shuffled-corruption ablation on recovery. The gap should grow with distance from the training-depth boundary.
- Transfer to natural data. Gains in $\rho$ on fictional data should come with gains on 3-hop and 4-hop MuSiQue and GRADE items, with no loss on 2-hop items. If gains appear only on fictional data, the agent has learned the generator's templates rather than recovery. We guard against this further by holding out paraphrase templates and relation types at test time.
References
- On Error Propagation of Diffusion Models
- Test-Time Iterative Error Correction for Efficient Diffusion Models
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Chain-of-Retrieval Augmented Generation
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
- GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluation
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries