When a language model answers a question that needs several lookups ("In what city was the director of Inception born?"), it runs a chain: search, read, extract a bridge fact, search again. If the final answer is wrong, we want to know which link broke first. We also want to know whether it broke because the search returned the wrong documents or because the model misread documents that were fine. The natural tool is an LLM judge, meaning a strong model prompted to grade another model's work. Two findings suggest that judges are least reliable exactly where this job is hardest. On long math solutions, GPT-4o scores only 61.9 F1 at locating the earliest erroneous step ProcessBench. Models that solve hard problems also fail easier ones when those are wrapped in redundant or distracting context Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?. A multi-hop retrieval trace is long, and much of its text is distracting retrieved material.
Multi-hop questions have a property that math proofs lack. Each hop is, by construction, an easy single-hop question answered by a short piece of evidence. That is literally how MuSiQue builds its questions. We propose to exploit that structure. Split the trace at its search calls. Show the judge one hop at a time, containing only that hop's sub-question, the passages retrieved for it, and the sub-answer the model stated. Ask two narrow questions: did the passages contain the answer, and does the stated sub-answer follow from them? The first hop that fails either check is the first error, and which check it fails labels the error as a retrieval failure or a reasoning failure. The bet is that turning one hard, long-context judgment into several easy, short ones makes an ordinary judge localize errors about as well as an expensive reasoning model reading the whole trace, at lower cost and without gold answers.
Core Claim
We make two claims. The first is that full-trace judges lose first-error accuracy as the erroneous hop moves later in the chain and as the number of retrieved passages in view grows. This is a measurable consequence of the distraction and hard-to-easy failures documented for LLMs generally. The second is that isolating each hop removes most of that decline. The logic is simple. If hop 2 is reasoned correctly from hop 1's wrong bridge entity, the isolated judge marks hop 2 as locally valid and correctly blames hop 1. "First locally invalid hop" therefore matches "first error" for linear chains. Where it does not match, as in comparison questions that merge two branches, the failure itself is informative and we report it separately.
The idea borrows its rationale from When Splitting Makes Stronger, which shows that splitting a task into independent subtasks shrinks each decoding context and limits error propagation. That paper applies splitting to the solver. We apply it to the grader, and multi-hop QA is the rare setting where the subtasks really are independent enough for that argument to hold.
Research Program
The first deliverable is a first-error benchmark for multi-hop retrieval reasoning, which the file suggests does not exist. ProcessBench covers only math. AgentEval and REFLECT cover agent traces but do not separate retrieval from deduction. Labels can be produced mostly automatically, because MuSiQue, 2WikiMultiHopQA and HotpotQA supply gold sub-answers and supporting paragraphs.
We will collect traces from several search agents: IRCoT-style prompting IRCoT, an ESearch-style RL agent in the spirit of Erase to Improve, and an outcome-RL agent. Each trace is aligned to the gold hops. A hop receives one of four labels. It is a retrieval failure if its gold paragraph was not retrieved and the sub-answer is wrong. It is a reasoning failure if the paragraph was retrieved but the sub-answer is wrong. It is an unsupported-correct shortcut if the sub-answer is right but no supporting passage was retrieved; this is the pattern GTA-RAG found in outcome-trained agents. Otherwise it is valid. Traces with a correct final answer but an earlier invalid hop are the multi-hop analogue of ProcessBench's finding that 51.8% of correct Omni-MATH solutions contain process errors. We will measure how common they are.
Automatic alignment will be imperfect. A stratified sample of about 1,000 traces will therefore be labeled by two or three annotators each. ProcessBench discarded 30% of its initial data for lack of annotator consensus, so we plan for comparable attrition and report agreement.
The second deliverable is the hop-isolated judge itself. We will prompt open judges (Qwen2.5-72B, and a small 7B judge to test whether isolation lets weak judges catch up) and a reasoning model. Each gets the two narrow questions per hop and a fixed short rubric. It never sees the original question, the final answer, or other hops, which also removes the answer-anchoring shortcut described in The Silent Judge.
The third deliverable tests whether better localization matters downstream, in two uses. First, we filter self-training data. We rerun a GRAT-style pipeline that keeps correct-answer traces, with and without discarding traces the judge flags. Second, we use the judge's flag at inference to trigger erase-and-retry on the offending hop. This replaces the gold-evidence trigger that Erase to Improve uses in training.
Baselines
The naive reference point is outcome-only. It predicts "no error" for correct-answer traces and "last hop" for wrong ones. Its score shows how much of the benchmark is solvable without reading the reasoning at all.
The strongest existing competitor is a reasoning model judging the whole trace, following ProcessBench's critic protocol. On math, o1-mini reaches 87.9 average F1 there, against 71.5 for QwQ-32B-Preview and 61.9 for GPT-4o ProcessBench. This is the baseline most likely to make our method unnecessary. We found no reported result on Lacuna for any judge on multi-hop retrieval first-error localization, so the numbers on our benchmark will be new. For a fair comparison, the full-trace reasoning judge and our isolated judges run on identical traces. We report accuracy against total judge tokens. Our method counts every per-hop call, and the full-trace baseline may use self-consistency voting until its token count matches ours. We beat it if a non-reasoning judge with isolation matches it within two F1 points at lower token cost, or if a reasoning judge with isolation exceeds it on hops three and later.
The second competitor is AgentEval-style step-level judging, where the judge grades each step but sees the full upstream prefix. AgentEval reports 72% root-cause accuracy against a human ceiling of 81%, and failure-detection recall of 0.89 versus 0.41 for end-to-end evaluation AgentEval. Run with the same judge model and rubric as ours, this is our key ablation. The only difference is whether the prefix and the other hops are visible, so any gap isolates the contribution of isolation.
The third is rollout-based localization, following Self-Explore: continuations are sampled from each prefix, and the first prefix after which success collapses is taken as the error. We give it the same token budget.
As an oracle ceiling we include gold-matching, meaning TF-IDF against gold evidence and sub-answer F1 as in Erase to Improve. It is unavailable at deployment and shows how much headroom remains.
A second ablation merges the two narrow questions into one "is this hop correct?" query. This tests whether separating retrieval from reasoning checks helps beyond isolation.
Downstream, the references are:
- the GRAT-style correctness filter, which moved Llama3-8B from 60.6% to 66.8% on 2WikiMultiHopQA GRAT;
- ESearch's reported gains of +8.48 EM and +11.56 F1 at 3B Erase to Improve;
- GTA-RAG's rise in full-chain evidence coverage from 54.3% under answer-only RL to 74.1% GTA-RAG.
Why not just…?
Why not just use a reasoning model on the whole trace? This is the real threat. ProcessBench shows reasoning models far ahead of standard judges, and they may already handle long retrieval traces well. If the full-trace reasoning judge's accuracy stays flat across hop position and passage count, our first claim is false. Isolation would then only be a cost saving, and we would report it that way. The position-and-distractor curves in the decisive experiment settle this directly.
Why not just grade each step with the full prefix, as AgentEval does? That design works for agent workflows, but its judge still reads every retrieved passage upstream. The distraction effect we target predicts that it degrades on later hops. The prefix-visible versus isolated ablation, with everything else fixed, answers this. If the two tie, isolation adds nothing and the benchmark becomes the main contribution.
Why not just use rollouts to find where success collapses? In multi-hop QA, rollouts can succeed from a broken prefix by recalling the answer from memory. That is exactly the shortcut behavior GTA-RAG documents, and it would mislabel the true first error as harmless. Rollouts also cost many continuations per hop. We will measure how often rollout labels disagree with human labels on unsupported-correct shortcut hops specifically.
Why not just train on final-answer correctness and skip localization? Outcome rewards leave nearly half of evidence chains incomplete (54.3% coverage in GTA-RAG). Sequence-level credit also punishes correct early steps Step-DPO. The self-training experiment checks whether this matters for accuracy and not only for faithfulness.
Decisive Experiments
The central experiment plots first-error F1 against the index of the true first error (hop 1 through 4, using MuSiQue's 2–4-hop questions) and against the number of retrieved passages per hop, which we vary by padding with retrieved distractors. We compute this for every judge in both full-trace and isolated modes.
We expect three outcomes if the claims hold. The full-trace curves should fall with hop index and padding. The isolated curves should be nearly flat. The 7B isolated judge should close most of the gap to the full-trace reasoning judge.
Failure is equally clear. If isolated and full-trace curves are indistinguishable, or if isolation badly hurts comparison and intersection questions whose hops are not independent, then the premise that multi-hop structure makes judging decomposable is wrong. We break results down by question type to test this.
The retrieval-versus-reasoning split is scored against the automatic and human labels. We give particular attention to shortcut hops, which neither outcome signals nor rollouts can see.
Finally, the two downstream uses are run on Qwen2.5-7B with matched training data and search budgets:
- keeping only judge-clean traces for self-training;
- judge-triggered erase-and-retry at inference.
Both are evaluated on HotpotQA, 2WikiMultiHopQA, MuSiQue and Bamboogle, using exact match, F1 and full-chain evidence coverage. We count the method as useful if it raises coverage without lowering exact match relative to the correctness filter, and if it recovers a substantial part of ESearch's gold-triggered gains without gold labels.
References
- ProcessBench: Identifying Process Errors in Mathematical Reasoning
- Can Large Language Models Always Solve Easy Problems if They Can Solve Harder Ones?
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Erase to Improve: Erasable Reinforcement Learning for Search-Augmented LLMs
- GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- GRAT: Guiding Retrieval-Augmented Reasoning through Process Rewards Tree Search
- Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards
- Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs