When a language model answers a question that needs several facts chained together, its mistakes are hard to catch. A typical example is "Who directed the film that won the award the novelist received?" A hallucination, meaning a fluent statement that is not actually supported, may hide in a premise the model claims to have read, in an inference that does not follow, or in an inference dressed up as a quoted fact. Today's automatic evaluators are usually another large model acting as a grader, called an "LLM-as-a-judge". They either grade the whole answer at once or check each sentence against all available evidence. Both approaches give the judge too much room. It can re-derive the entire chain in its own head and inherit the same depth-driven collapse that afflicts the model being graded. It can also quietly accept a step because the step is true according to something, even though it is not true according to what the step claims as its support.
This note proposes a judge that first labels what each step of an answer is doing and then checks the step only against the support that its role allows. The labeling borrows from rhetorical role labeling, the older NLP task of tagging sentences by communicative function (background, method, result) rather than by topic. There are four roles. A "premise from source" must be entailed by the cited passage alone. An "inference" must follow from earlier steps alone. An "assumption" must be marked as tentative. The "conclusion" must follow from its parents. The bet is twofold. Each check becomes short and local, so judge accuracy should stay roughly flat as the number of hops grows. Separately, a specific and under-measured class of hallucination becomes detectable: steps whose stated warrant does not match their real one.
Core Claim
The claim has two parts, and each can be tested on its own. The first part concerns depth. A judge that verifies a k-hop answer holistically must track all k dependencies at once. The multi-hop literature shows this degrades sharply with depth. Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples reports accuracy collapsing between proof depth 2 and 5 on synthetic ontologies. DyVal shows that sequential depth hurts far more than parallel breadth. If per-step success compounds multiplicatively, as RuleBERT argues (0.9 per step gives about 0.53 over six steps), then a judge that reasons through the whole chain should decay the same way. A judge whose every call checks a single step against a small, role-determined context should not decay that way.
The second part concerns warrants. Some errors are invisible to any checker that asks only whether a step is supported by the documents or by the earlier steps, with either source counting. Two examples follow. In the first, an inference is written as though a cited source states it. This is fabricated attribution, and it is also the "tentative finding becomes definitive fact" pattern in Liyan Tang et al.. In the second, a bridge "inference" quietly introduces a new fact from the model's own memory that neither the documents nor the prior steps contain. Making warrants asymmetric is what catches both. A premise step may look only at documents, and an inference step may look only at its parent steps.
How This Differs From the Closest Work
AgentEval motivates and supplies much of the method. It grades agent traces as a dependency graph with a GPT-4o judge and finds that 63% of step failures propagate from upstream. Step-level grading raised failure-detection recall from 0.41 to 0.89. Its nodes, however, are typed by operational function (planning, tool use, synthesis). They are not typed by what kind of support a natural-language claim requires. It also does not study factual hallucination.
Halu-J trains a 7B critique judge that runs four fixed procedural phases over retrieved evidence. It reaches 91% on ME-FEVER against 83% for GPT-4o. Its phases describe the judge's procedure, not the structure of the answer, and it verifies essentially flat claims. It does not examine chains deeper than two hops.
Improving Attributed Long-form Question Answering with Intent Awareness shows that making a generator state why it cites each source improves citation precision. It reports gains of +18.7 points for small models and +3.7 for frontier models. It acts on the generator, not the judge, and it does not model multi-hop dependencies.
Co-opNet constrains the design. Its span-matching factuality discriminator reduced hallucination but pushed summaries toward verbatim copying. This is a warning that any checker that requires every sentence to be found in the source will penalize legitimate inference. Role typing is meant to avoid exactly that, by checking inferences for validity rather than for presence in the source.
HaluEval raises a tension that our design must explain rather than ignore. Chain-of-thought prompting (asking the judge to reason step by step before deciding) degraded ChatGPT's hallucination detection on its HotpotQA-seeded QA subset. Meanwhile, Evaluating Generative AI with Large Language Models reports chain-of-thought judges approaching human agreement. Our hypothesis is that chain-of-thought hurts when the judge's own reasoning must re-traverse a long dependency chain, and helps when that reasoning is short. Role-local checks keep it short, so this proposal is also a direct test of that explanation.
Research Program
The system has three parts.
The first is a step segmenter and role labeler. It splits a model's multi-hop answer into steps, assigns each step one of the four roles, and records which earlier steps or cited passages the step claims to depend on. The schema is deliberately small. Fine schemas such as the 15-category AZ-II reach only κ = 0.65–0.71, while the collapsed 7-category version reaches 0.75 (Towards Domain-Independent Argumentative Zoning). The Fact/Evaluation split in AMPERE reached κ = 0.93, which suggests that "is this asserted from a source or from reasoning" is a distinction humans make reliably. We will measure human agreement on our own four-way scheme before trusting it.
The second part is the role-conditioned verifier. A premise step is shown only its cited passage and asked whether the passage entails the claim. An inference step is shown only its parent steps and asked whether the claim follows, with any new factual content flagged. An assumption is checked for hedging. The conclusion is checked against its parents.
The third part is error propagation. Following AgentEval, a step that is valid given its parents but has an invalid ancestor is marked as inheriting the error. This lets us localize the root hop.
The analysis program asks where errors in natural multi-hop answers concentrate by role transition. Gap 1 of the research file notes that nobody has measured whether decay clusters at, for example, transitions from premise to bridge inference. It also asks whether the judge's accuracy slope against hop count differs between holistic, untyped-stepwise, and role-typed judging.
Baselines
The naive reference point is final-answer checking: exact match or F1 of the final answer against gold, which ignores the chain entirely. This gives the false-acceptance rate for answers that reach the right entity through an invalid or fabricated chain. That is the "reward tampering" case flagged by AutoPRM, and it is the shortcut behaviour that MuSiQue was built to expose. We expect this baseline to miss nearly all hallucinations hidden inside correct-answer chains.
The second baseline is a holistic judge in the style of HaluEval. The same base model (GPT-4o and one open model) is shown the question, all documents, and the full answer, and returns a hallucinated/not label and the offending step. It is run with and without chain-of-thought, because HaluEval's result makes both variants necessary. HaluEval reports 62.59% QA detection accuracy for ChatGPT without knowledge and 76.83% with knowledge. That is the closest reported figure on HotpotQA-derived data, though it measures answer-level labels and not step localization.
The third baseline is Halu-J's four-phase critique pipeline. We will use released weights if they are available; otherwise we will reproduce the four-phase prompt on the same base model. Its reported 91% accuracy against GPT-4o's 83% on ME-FEVER makes it the strongest specialized hallucination judge in the file.
The strongest competitor on this specific question is an untyped stepwise judge modeled on AgentEval. It uses the same segmentation and dependency graph as our system, but every step is shown both its cited passages and its parents and asked whether it is supported. This is the baseline that most threatens the proposal, because it already has locality and propagation tracking. The only thing it lacks is warrant asymmetry. AgentEval reports failure-detection recall of 0.89 with κ = 0.84 against humans on its agent traces. There is no reported result on Lacuna for any step-level judge on MuSiQue or PRONTOQA-OOD.
Fairness rules apply to all baselines. Every judge uses the same base model, the same documents, and the same segmentation where segmentation applies. All prompts get the same prompt-tuning budget on a shared development split. Total judge tokens per answer are reported, and we include a compute-matched holistic variant that samples several holistic judgments and takes a majority, so that the stepwise methods do not win merely by spending more calls.
Two ablations isolate the key ingredient. The first gives role labels to the untyped judge but keeps full context, which separates "knowing the role" from "restricting the warrant." The second compares oracle role labels with predicted ones, which measures how much labeler error costs.
Beating the strongest baseline means two things. First, higher step-level recall on warrant-mismatch errors at equal precision. Second, a flatter accuracy-versus-hop slope, while the false-positive rate on valid inferences stays no higher than the untyped judge's. A win on recall bought by flagging legitimate inferences would just reproduce Co-opNet's extractive trap and would not count.
Why not just…?
Why not just use an untyped step-level judge like AgentEval's? This is the real threat, and the proposal is built around testing it rather than assuming it away. The untyped judge's acceptance rule is "supported by the documents or by the parents." By construction, that rule passes a true inference falsely attributed to a source. It also passes a bridge fact the judge happens to know from memory. Whether this matters in practice is empirical. If the injected warrant-mismatch errors turn out to be rare in natural outputs, or the untyped judge catches them anyway, then role typing adds nothing beyond AgentEval and we will report that. The decisive comparison is the full-context-with-roles ablation against the restricted-warrant system.
Why not just ask a frontier model to grade the whole answer with chain-of-thought? HaluEval found that chain-of-thought lowered detection accuracy on its multi-hop QA subset. AgentEval found that end-to-end checking recovered under half the failures that step-level checking found. Neither paper tested whether the cause is the judge's own depth decay. Our depth-stratified experiment tests exactly that. If holistic accuracy stays flat across 2 to 4 hops, the depth half of our claim is wrong.
Why not just decompose answers into atomic facts and check each against the retrieved documents? This is the logic behind checkers like FActScore, which the hallucination survey lists. Valid inferences are by definition not stated in any document, so such a checker must either flag them, which reproduces the copying pressure seen in Co-opNet, or exempt them, which leaves invalid leaps unchecked. It also misses entity and role swaps where every entity appears in the source, as in Léo Milecki. We include it as an additional baseline and report its false-positive rate on valid inferences specifically.
Why not just train a process reward model? Procedural supervision of the kind AutoPRM uses improves the generator, and its advantage over chain-of-thought widens with chain length. It does not tell us whether a step's claimed warrant is its real one, and it requires training on task-specific data. Our verifier is training-free. If it works, its role-local verdicts could later serve as labels for such a model, so the two approaches complement rather than compete.
Proposed Experiments
The first testbed is MuSiQue, chosen because its questions are composed bottom-up from single-hop questions with gold decompositions. That gives a reference chain for 2-, 3-, and 4-hop items. It also filters out disconnected shortcuts, which caused a 30-point F1 drop for single-hop models relative to HotpotQA. Human performance is 78.0 answer F1 and 93.9 support F1. We generate attributed chains from several models and use the gold decomposition to inject single errors at a controlled hop position. There are four error types: fabricated premise, invalid bridge inference, warrant inflation (an inference attributed to a passage that does not state it), and entity swap. Each corrupted chain has a matched clean twin. We also hand-annotate a subset of uncorrupted natural outputs, so that the error mix is not purely synthetic. That subset is where the role-transition analysis of naturally occurring decay is done.
The second testbed is PRONTOQA-OOD. There, step validity is checkable by program and depth runs from 1 to 5 without memorization confounds, so the judge's depth slope can be measured cleanly. The third is the QA subset of HaluEval, used to check that our answer-level verdicts remain competitive where prior numbers exist.
Metrics are step-level precision and recall per error type, root-hop localization accuracy, and the slope of detection accuracy against hop count. We also report the false-positive rate on valid inferences, judge tokens per answer, and agreement with human step labels (Cohen's κ).
Three checks decide the outcome. First, on PRONTOQA-OOD and MuSiQue, does the holistic judge's accuracy fall with depth while the role-typed judge's does not? Second, on warrant-inflation and memory-bridge errors, does the restricted-warrant system beat the untyped stepwise judge, and does the gain vanish in the full-context-with-roles ablation? A gain that vanishes there would show that the restriction, not the label, does the work. Third, does chain-of-thought's effect on the holistic judge flip from harmful to neutral as chains shorten? That would be direct evidence for the depth explanation of HaluEval's chain-of-thought result.
If the first check fails, the depth claim is wrong. If the second fails, role typing is redundant with AgentEval-style grading.
References
- Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- RuleBERT: Teaching Soft Rules to Pre-Trained Language Models
- Evaluating large language models on medical evidence summarization
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Halu-J: Critique-Based Hallucination Judge
- Improving Attributed Long-form Question Answering with Intent Awareness
- Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- Towards Domain-Independent Argumentative Zoning: Evidence from Chemistry and Computational Linguistics
- Argument Mining for Understanding Peer Reviews
- AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
- MEDIMP: 3D Medical Images and clinical Prompts for renal transplant representation learning