Judging Multi-Hop Answers by Checking the Chain They Claim

Language models are now routinely used as graders for other models. This practice is called "LLM-as-a-judge." These graders break down on questions whose answer depends on a chain of facts, such as "In what city was the director of Inception born?" A judge that reads the final answer cannot tell whether "London" is right unless it already knows both links. When pairs of answers are stripped of style differences, strong judges score near coin-flip. Meanwhile, "rubric" methods, which have the judge first write a checklist of criteria for each prompt, are written once from the question alone. For multi-hop questions that is the wrong moment. The facts that need checking, such as the intermediate person "Christopher Nolan," are not visible in the question. They only appear in the answer being graded or in evidence found along the way.

We propose that the judge build its checklist from the candidate's own claimed reasoning chain. Each claimed link becomes one checklist item. The judge verifies each item with a retrieval query aimed at that link, and it scores each item without seeing the rest of the answer. A separate check asks whether the verified links actually connect back to the question. The final verdict is computed by a fixed rule from these per-link outcomes, not by the judge's free-form impression. The bet is that verifying a claimed chain is easier than solving the question from scratch, so this judge can escape the "a judge is only as good as it is as a solver" ceiling. A second bet is that computing verdicts from isolated, evidence-grounded checks removes the openings that shortcut biases such as provenance labels, recency labels and verbosity currently exploit.

Core Claim

Three bodies of evidence motivate this. On correctness-focused pairs, JudgeBench shows vanilla GPT-4o at 50.86% pairwise accuracy, and it reports that judging ability tracks problem-solving ability. IRCoT shows that for answering multi-hop questions, using each intermediate reasoning sentence as the next search query improves retrieval recall by 11–21 points over retrieving once up front. Rubric-Grounded RL shows that a judge given the ground-truth source passage, which the policy never sees, gives much more useful rewards. In open-domain multi-hop settings, however, nobody hands the judge that passage.

Our claim joins these three findings. A judge can build its own verification key by treating the candidate's intermediate claims as retrieval queries, the way IRCoT does for answering. That key turns an unsolvable holistic judgment into a set of small, independently checkable ones. What is new is where the rubric comes from and when it is made. The rubric is generated from the response's claimed chain, and it grows by one item each time a claimed link is refuted and retrieval turns up the correct link. It is not generated once from the prompt, and it is not taken from a fixed failure taxonomy.

Relation to Prior Work

Several dynamic rubric methods generate criteria per prompt but fix them before evidence arrives. Recursive Rubric Decomposition (RRD) splits broad criteria until they discriminate between responses and down-weights correlated items, lifting GPT-4o on JudgeBench from 55.6% to 73.3%. GenRubric trains rubric generators with reinforcement learning. EvalPlanner writes an evaluation plan before reading responses. AdaRubric writes five dimensions per agent task and keeps them fixed across the trajectory. All four produce flat criteria from the prompt, with no retrieval and no representation of which claim depends on which. We supply both.

Two lines of step-level evaluation come closest. AgentEval grades each agent step against its upstream context and raises failure recall from 0.41 to 0.89, but it grades against a fixed 21-category taxonomy. Step-wise Rubric Rewards attaches pre-distilled rubric items to math reasoning steps, with no retrieval. We borrow AgentEval's idea of judging each node against its parents, but the nodes and their criteria come from the response itself.

Two papers act as constraints on the design. Local Coherence or Global Validity? finds that GPT-4o tags logical errors at only 57.8% accuracy and mistakes locally fluent steps for sound ones. This tells us per-step fluency checks are not enough, and it is why every item must be decided by retrieved evidence plus an explicit test that the chain connects to the question. Judging LLM-as-a-Judge shows two things: a classifier reading only the rubric text predicts verdicts above 0.80 balanced accuracy, and judges often fail to flip their verdict when the meaning of an output or a rubric is inverted. Checklists generated from the response are especially exposed to this "verdict leaks through rubric wording" risk. We therefore make that classifier test a first-class measurement rather than an afterthought.

Finally, The Silent Judge and LLMBar supply both the bias tests and the reason not to rely on free-form chain-of-thought. The Silent Judge reports verdict shifts of up to +30% from recency labels, with 0% acknowledgment of the cue in the judge's rationale. LLMBar finds that chain-of-thought lets judges rationalize polished answers that violate the instructions.

Research Program

The system runs in four stages for each candidate answer.

First, an extraction step rewrites the response as a small graph. Each node is an atomic claim linking two entities, such as "Inception → director → Christopher Nolan." Each edge records that one claim's entity feeds the next. If a response gives only a bare answer, the graph is a single node.

Second, each node becomes a binary checklist item. That item is checked by a verifier that sees only the claim, retrieved passages, and the parent claims it depends on. The verifier never sees the candidate's prose, formatting, metadata or final verdict. Retrieval for each node uses the claim itself as the query, which is the IRCoT move applied to grading.

Third, when a node is refuted, the verifier states what the evidence says instead. That corrected link spawns a new checklist item for the downstream hop. This lets the judge detect the case where a later claim is true but hangs off a wrong parent. A connectivity check then asks whether the chain of verified links, read from the question's entities, actually terminates at the stated answer. This targets the disconnected-shortcut failure that MuSiQue was built to expose.

Fourth, the verdict is a fixed rule over node outcomes. A candidate receives credit for its answer only if a connected, fully verified path supports it. In pairwise mode, the pair is compared by that rule, with the count of verified hops breaking ties.

We would build this with an open backbone, Llama-3.1-70B or Qwen3-32B, so it can be run with and without training. We would add a small trained variant in which the extractor and verifier are fine-tuned on MuSiQue's gold single-hop decompositions.

The evaluation set is a new judging benchmark built from MuSiQue-Ans, 2WikiMultihopQA and HotpotQA. Following JudgeBench, both answers in each pair come from the same generator model, so style is neutralized. We label each pair by correctness. Because MuSiQue questions are composed from single-hop questions, we can also label which hop is wrong. We deliberately include "right answer, broken chain" and "every hop true but disconnected" cases.

Baselines

The naive reference point is a vanilla judge prompt that reads the question and both answers and picks one. On JudgeBench, JudgeBench reports 50.86% for GPT-4o, 56.57% with the Arena-Hard prompt, and 64.29% for Claude-3.5-Sonnet. We rerun this prompt with our backbone on every test set.

The strongest existing rubric method is RRD with whitened weighting. The paper reports 73.3% pairwise accuracy on JudgeBench with GPT-4o. It is the strongest competitor because it already turns holistic grading into many discriminative criteria. We run RRD with the same backbone and the same number of response samples for decomposition, and we allow it the same total model-call budget as our method.

The strongest reasoning-heavy reference is o3-mini at high reasoning effort, reported at 80.86% overall on JudgeBench JudgeBench. We report it as a ceiling on what extra thinking alone buys.

No reported result was found on Lacuna for any LLM judge on a multi-hop QA judging benchmark derived from MuSiQue, HotpotQA or 2WikiMultihopQA. All numbers on our new set will therefore be rerun baselines.

Two retrieval baselines isolate the role of retrieval. The first is one-shot retrieval with a static rubric: top-k passages for the question, graded against a fixed Likert rubric for accuracy and completeness. The second is solve-then-judge. Here the judge first answers the question itself with IRCoT, then marks each candidate by agreement with its own answer. Both get the same retriever, corpus, and retrieval-call budget as our method.

Two ablations isolate the key ingredients. A flat claim-check variant verifies the same extracted claims, but with no dependency edges, no branch spawning, and no connectivity test. A non-blinded variant lets the verifier see the whole response.

An oracle upper bound gives the judge MuSiQue's gold supporting paragraphs, as in the information-asymmetric setup of Rubric-Grounded RL.

Beating the field means three things. On the new multi-hop set, we must exceed both RRD and solve-then-judge at matched budget and close a substantial fraction of the gap to the oracle. On JudgeBench's Knowledge subset, we must match or exceed RRD. The overall 73.3% figure mixes in math and coding, where retrieval should not help, so we report that subset split separately and do not claim gains there.

Why not just…?

Why not just have the judge solve the question itself and compare answers? This is the most serious threat, because JudgeBench's finding that judging ability tracks solving ability suggests grading may reduce to solving. Our counter-hypothesis is that verification is easier than search. The candidate's intermediate entity, such as "Nolan," is exactly the query a solver lacks, and IRCoT's gains come precisely from having that intermediate text available as a query. The solve-then-judge baseline tests this directly at equal retrieval budget. The analysis we care most about is on questions where the IRCoT solver itself fails. If our judge's accuracy there does not beat solve-then-judge, the idea reduces to re-solving and we would say so.

Why not just give the judge the gold supporting passages? When references exist, that is the right choice, and our oracle condition measures it. The proposal targets the common case where no reference exists: open-domain answers, reward models scoring fresh policy outputs, and deployed retrieval-augmented systems. The question is how much of the oracle's advantage a judge can recover by building its own key.

Why not just decompose the answer into atomic facts and check each one? Flat fact-checking misses answers whose facts are individually true but do not form a chain from the question to the answer. This is exactly the shortcut that MuSiQue's disconnection filters were designed to remove. The flat claim-check ablation settles this question: we expect it to match our method on single-wrong-hop pairs and to lose on the disconnected and wrong-parent pairs. If it does not lose, the graph structure is unnecessary.

Why not just add chain-of-thought or bias-warning instructions to the judge? LLMBar found that chain-of-thought helps judges rationalize polished but wrong answers, and The Silent Judge found 0% cue acknowledgment, which means rationales hide what actually drove the verdict. We include a "think carefully and ignore labels" prompt as a baseline in the bias tests.

Proposed Experiments and Decisive Checks

The first decisive check is accuracy on the new style-neutralized multi-hop judging set, broken out by failure type: wrong final hop, wrong middle hop, right answer with a broken chain, and disconnected chain. Here we also measure hop-level error localization against MuSiQue's gold decompositions, and whether localization transfers to 2WikiMultihopQA and HotpotQA.

The second check is bias robustness. Following The Silent Judge, we inject provenance labels ("EXPERT" versus "UNKNOWN"), recency labels, and verbosity or markdown padding into one answer of each pair. We then measure the verdict shift rate for every system. We predict our method shifts far less than both the vanilla judge and RRD, because cues never reach the blinded verifier. The non-blinded ablation tells us how much of that robustness comes from blinding rather than from retrieval.

The third check applies the rubric-text probe from Judging LLM-as-a-Judge to our generated checklists. We train a classifier on checklist text alone to predict verdicts; a result near chance means the verdict is not leaking through the wording. We also apply the paper's inversion test, flipping a claim's meaning and checking whether the verdict flips.

The fourth check asks whether the gains transfer to existing benchmarks. We measure accuracy on JudgeBench's 154 Knowledge pairs against RRD. On LLMBar's adversarial set, where retrieval is irrelevant, we check that the method does no harm relative to the "Rules + Metrics + Reference" result of 82.8% reported in LLMBar.

Throughout, we report cost as model calls and retrieval calls per judgment.