Telling Expert Inference Apart from Hallucination in Retrieval-Grounded Answers

Systems that answer questions by first fetching documents and then writing a response are now routinely checked by a second language model acting as a grader. That grader asks whether each claim in the answer is supported by the fetched text, and it flags anything else as a fabrication. The rule works for summarizing news. It breaks in expert domains, where the most valuable claims are often ones the text never states. A physician who reads "lactate 4.8, temperature 39.1, heart rate 124" and writes "sepsis" is not making something up, even though the word appears nowhere in the note. Current graders force a bad choice. They either call such inferences hallucinations, which punishes the reasoning domain experts actually need, or they let the grader consult its own memory, which is precisely the memory that fails in specialized domains.

We propose a grader that gives a claim the status "derivable" only when it can produce an explicit, checkable argument for it. Each premise of the argument must be found in the given context or in a separately retrieved body of domain knowledge, such as clinical guidelines or textbook theorems, and each step must be verified on its own. Claims with such an argument are kept apart from claims stated outright and from claims with no traceable basis. We test the grader where the stakes are clearest: emergency diagnosis from admission data, clinical hallucination traps, and theorem-based STEM problems. The decisive test uses paired examples in which deleting a single lab value from the context should turn a valid inference into a fabrication. A good grader must flip its verdict on exactly those pairs.

Core Claim

Existing hallucination graders for retrieval-augmented generation (RAG), meaning systems that retrieve documents and condition generation on them, sit at two poles. At the strict pole are RLSeek, which trains a detector with reinforcement learning to quote verbatim source evidence before judging and penalizes quotes that do not lexically match the source, and REFCHECKER, which splits answers into subject–predicate–object triplets and labels each as entailed, contradicted, or neutral against the reference. Both treat "not in the text" as suspect. The REFCHECKER paper also documents the problem at the other pole: graders show "internal knowledge bias", quietly using their own memory instead of the reference. FaithJudge sits between the two. It calibrates a grader with human-labeled examples that include a "benign" category, meaning additions supported by world knowledge but absent from the source. The authors leave the boundary of that category unresolved and test only on general-domain text.

Our claim is that the benign/unwanted boundary becomes tractable and auditable once "benign" is replaced by "derivable with a verified argument." The argument has to satisfy three conditions. Its leaves must be facts stated in the context or statements from a retrieved domain-knowledge source. Every step must pass an independent entailment check. The conclusion must match the claim. A claim that sounds medically plausible but rests on a lab value the patient never had will fail at the leaves. This is the case that both a lenient memory-based grader and a "retrieve more, then check entailment" grader are expected to miss.

The need is documented in the domain literature. EarlyDx shows that frontier models recover only 3% to 31% of implicit diagnoses, those inferable from vitals and labs but not written in notes. A grader that penalizes unstated claims would therefore score the better model worse. KMQA finds that clinical exam questions require pathophysiological principles missing from the passage. TheoremQA constrains the design: pasting theorem definitions into the prompt did not help models solve problems. That result warns us that retrieving domain knowledge is not enough on its own. The grader has to show how a principle applies to the case, which is why we require a stepwise argument rather than co-presence of evidence.

Research Program

The grader runs in four stages. The first decomposes the answer into atomic claims, reusing REFCHECKER's triplet decomposition as the method. The second checks each claim for direct support in the context, which yields the "stated" label. For each unsupported claim, the third stage retrieves from a fixed domain corpus using hybrid sparse and dense retrieval. In medicine the corpus is guideline and textbook text. For STEM it is the TheoremQA theorem statements. A reasoning model then writes a short argument whose premises are explicit quotations or pointers into either the context or the retrieved passages. The fourth stage re-checks each premise against its cited source. Unlike RLSeek's lexical-overlap check, this check allows numeric and terminological normalization, so that "HR 124" matches "tachycardia >100". Each step is then checked by a separate entailment call that sees only that step's premises, which prevents the checker from relying on memory.

The output is one of four labels: stated, derivable, contradicted, or baseless. Every derivable verdict comes with its argument, which a domain expert can audit in seconds. HalluGraph's knowledge-graph alignment for legal RAG (HalluGraph) gives similar auditability, but it scores only entity and relation overlap with the context. It therefore cannot credit inferences, and its authors report collapse on short contexts.

Building the evaluation data is half the work. From EarlyDx encounters we collect candidate diagnoses from several generators. EarlyDx's own auditor labels (supported, partially supported, unsupported) and its explicit/implicit split give silver labels. Physicians adjudicate a stratified sample of roughly 1,500 claims. We then build paired counterfactuals in the spirit of the condition-removal used by ACA-RL. For each derivable claim, we delete or alter the key premise in the admission context, for example by removing the lactate or normalizing the temperature, and physicians confirm that the claim is no longer warranted. We create analogous pairs for TheoremQA solution steps by removing a given quantity. From MHB we take its adversarial "hallucination traps", which bait models into fabrication, and place the cases in a RAG setting with retrieved guidelines. Finally, we keep FaithBench as a general-domain check so that gains in medicine do not come from becoming lenient overall.

Baselines

The naive reference point is a strict context-only grader that labels every claim not entailed by the context as hallucinated. It is implemented as REFCHECKER's triplet checker with neutral counted as a hallucination. By construction it labels all derivable claims as hallucinated. It sets the floor for recall on derivable claims and the ceiling for flip consistency on baseless claims.

The strongest existing method is FaithJudge (Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards). It is a reasoning-model grader prompted with human-annotated examples labeled unwanted, benign, or questionable, and its authors report balanced accuracy of 84% and macro-F1 of 82.1% on FaithBench using o3-mini-high. It is the strongest competitor because it is the only method in the file that explicitly tries to admit knowledge-supported additions. We will run it with the same backbone model as our grader, the same context, and in-context examples drawn from our training split, mapping its benign label to derivable. To beat it, we must match it on FaithBench within noise and clearly exceed it on the paired-counterfactual flip rate in medicine. Without the flip-rate gain we have no result.

We also compare to RLSeek (RLSeek) as the strongest trained strict detector, and to the fine-tuned RAGTruth critic (RAGTruth). The RAGTruth paper reports a response-level F1 of 78.7% for its critic, against 63.4% for GPT-4 prompting. Both are run on our splits under the same annotation budget. For the clinical setting we include an EarlyDx-style instruction-tuned judge. No reported judge-agreement number was found on Lacuna for EarlyDx's Qwen3.5-27B judge, so we measure its agreement ourselves.

The key ablation is "retrieval without argument". The grader receives the same retrieved domain passages and makes a single entailment call over context plus passages, with no stepwise argument and no premise re-check. This isolates whether the structured argument, rather than the extra knowledge, does the work.

Why not just…?

Why not just prompt a strong reasoning grader to accept claims supported by medical knowledge? This is the real threat, since frontier models know a lot of medicine and FaithJudge already does a version of it. The weakness lies in the cases that matter. A grader that consults its own memory will accept "sepsis" whether or not the lactate is actually in the note, because the claim is plausible. REFCHECKER's internal-knowledge-bias finding predicts exactly this. The paired-counterfactual set measures it directly: a memory-based grader should label both members of a pair derivable. If the prompted grader flips correctly on pairs about as often as ours does, we will report that the argument machinery is unnecessary.

Why not just give the grader retrieved guidelines and check entailment once? That is our main ablation. It may close much of the gap. Our expectation that it falls short comes from TheoremQA, where providing theorem text did not make models apply it correctly, and from the observation that single-call entailment over a long mixed context does not show which premise was used. If the ablation matches the full system on flip rate and expert agreement, the contribution shrinks to "retrieve domain knowledge for the grader", and we will say so.

Why not just fine-tune a detector on domain-labeled data? The RAGTruth critic shows that fine-tuned detectors beat prompted GPT-4 in-domain. However, the study of fine-tuned judges finds that such judges behave as task-specific classifiers that fail across schemes and aspects, and dense span-level clinical annotation is expensive. We test the option by fine-tuning a detector on EarlyDx silver labels and evaluating it on MHB-based cases and TheoremQA pairs, which tests transfer, not only in-domain fit.

Proposed Experiments

The primary metric is four-way macro-F1 against physician-adjudicated labels, with recall on derivable claims and precision on baseless claims reported separately. The decisive check is flip rate on counterfactual pairs: the fraction of pairs where the grader labels the original claim derivable and the premise-deleted version baseless. Strict graders score near zero by construction because they reject the original claim. Memory-based graders should also score near zero, because they accept both versions. We predict a clear separation for our grader. A second check is audit cost: physicians rate whether the shown argument is correct and how long verification takes, compared with judging the bare claim. A third check holds FaithBench performance against FaithJudge's reported 84% balanced accuracy to rule out global leniency. As a downstream test, we rerank sampled EarlyDx generations with each grader and measure implicit-diagnosis recall against the baseless-claim rate. This asks whether a grader that credits valid inference lets a generator say more true unstated things without saying more false ones.

The outcome that would falsify the proposal is that the retrieval-only ablation or prompted FaithJudge reaches comparable flip rates. In that case explicit arguments add auditability but not accuracy, and the note's claim reduces to that weaker one.