When a language model reads a patient's chart and fills in the facts that decide treatment, some of those facts will be made up. Examples are a kidney function value, a tumor stage, or a biomarker score. Current practice either checks everything, which is impossible at scale, or checks the outputs the model seems least sure about. Neither asks the question a health system cares about: which wrong fact would change a treatment decision enough to cost money or quality-adjusted life years (QALYs, years of life weighted by health quality)? This note proposes a rule for routing a fixed budget of clinician review time across extracted facts. Each claim is ranked by its expected loss in health-economic value if left unchecked, priced using the same willingness-to-pay threshold (the dollars per QALY a payer accepts, often $100,000–$150,000 in the US) that health technology assessment already uses to approve therapies.
The core idea fits in two sentences. The expected harm of a hallucinated fact equals the probability it is wrong times how much the resulting decision loses in "net monetary benefit," which is QALYs gained times the threshold, minus cost. Ranking facts by this product, rather than by hallucination probability alone, should let a small amount of clinician review recover most of the value lost to extraction errors, especially for facts sitting near eligibility cutoffs that gate expensive therapies.
Core Claim
Two literatures meet here without talking to each other. Health-economic evaluations of AI tools treat the tool as a test with fixed sensitivity and specificity. The analysis of AI kidney risk stratification against the standard KDIGO guideline (Cost-Effectiveness Analysis of Artificial Intelligence-Driven Risk Stratification in Patients With Diabetic Kidney Disease in the US Veterans Population) reports an ICER of $116,349/QALY and finds specificity to be the most sensitive parameter. It never asks where errors come from or whether they can be selectively caught. The hallucination-detection literature, on the other hand, prices errors only in compute. Cost-Effective Hallucination Detection for LLMs combines cheap calibrated scores under an API-call budget, and CEBench benchmarks accuracy per token dollar. In both, every hallucination costs the same.
Our claim is that both framings get the binding constraint wrong. Compute for verification costs cents per record, so it is not what limits safety. The scarce resource is clinician attention for auditing evidence. The right exchange rate for spending that attention is the cost-effectiveness threshold $\lambda$. For claim $j$ in patient $i$'s record, let $\hat x_{ij}$ be the extracted value and $p_{ij}$ the calibrated probability that it is wrong. Let $d(\cdot)$ be the clinical decision rule, such as treatment eligibility, and $\text{NMB}_i(d) = \lambda\, \Delta E_i(d) - \Delta C_i(d)$ the net monetary benefit of that decision from the decision model, with $\Delta E$ in QALYs and $\Delta C$ in dollars. The value of sending the claim to review is
$$V_{ij} = p_{ij}\; \mathbb{E}\big[\,\text{NMB}_i(d(x_{ij})) - \text{NMB}_i(d(\hat x_{ij})) \mid \hat x_{ij}\ \text{wrong}\,\big] \cdot q_{ij},$$
where $x_{ij}$ is the true value and $q_{ij}$ is the probability a reviewer catches the error when shown the model's quoted evidence. Given review times $t_{ij}$ and a total budget $T$ minutes per patient batch, we route the claims that maximize $\sum V_{ij}$ subject to $\sum t_{ij} \le T$. This is a knapsack problem, solved greedily by $V_{ij}/t_{ij}$.
$V_{ij}$ is the standard expected-value-of-information quantity from health economics. What is new is applying it per extracted claim to LLM outputs, with the probability term supplied by a hallucination detector and the catch term supplied by an interpretable evidence display. A side effect is that the threshold changes which errors matter. At a given $\lambda$, falsely declaring a patient eligible may waste money while falsely excluding one loses QALYs. Moving $\lambda$ reweights these errors, so the routing policy is tied to the payer's actual decision context.
Research Program
The system has three parts.
The first is the extractor, which is the source of hallucinations. We use the four-step Self-Verification pipeline of Self-Verification Improves Few-Shot Clinical Information Extraction: extract, look for omissions, quote a verbatim evidence span, then prune unsupported items. We reuse it as-is because its quoted spans overlap clinician-identified spans 93% of the time. That supplies the interpretable audit object a reviewer sees.
The second part is the error-probability estimate $p_{ij}$. We train a calibrated ensemble of cheap single-generation signals following Valentin et al.: token probability, self-assessed truth probability, and input-contradiction scoring. We add one signal adapted from The Hallucination Dependence Index. That paper compares a model's claims given the patient record against its claims given only a case ID. It shows that grounded support rates compress into 81.9–93.2% while ungrounded fabrication varies from 5.9% to 91.5%, so high grounded scores can hide recycling of what the model learned in pretraining. We turn its cohort-level diagnostic into a per-claim feature: semantic overlap between a grounded claim and the same model's ungrounded output for that case. High overlap suggests the value came from priors, not the chart. The cost is one extra call per patient.
The third part is the consequence term. It comes from re-implementing two published decision models. One is the kidney model from the AIKD analysis: a decision tree into low, medium, or high risk feeding a 5-year Markov model over CKD stages, dialysis, transplant, and death. The other is a simplified oncology eligibility model built on the pembrolizumab analysis (Cost Effectiveness of Pembrolizumab vs. Standard-of-Care Chemotherapy…). That model reports an incremental cost of $102,439 for 1.05 QALYs (ICER $97,621/QALY) when treatment is restricted to PD-L1 score ≥50%. We use the published parameters and propagate each claim-level error through the decision rule to a change in net monetary benefit. We do not invent parameters the papers do not report. Where the oncology model lacks a value for treating ineligible patients, we run a range in sensitivity analysis and say so.
The reviewer-catch term $q_{ij}$ and review time $t_{ij}$ are first set from simulation and then measured in a small clinician study. The study is motivated by the AICare user study (Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot). That study found that interpretable displays help clinicians catch errors, but that visible flawed reasoning sharply erodes trust, and that junior and senior clinicians use explanations differently. A routing policy that raises many low-stakes flags may therefore waste time and lose clinicians' engagement. We measure catch rate, time per claim, and trust as a function of flag volume.
Baselines
The naive reference is no routing. Self-Verification extractions go straight into the decision model, which gives the residual net-monetary-benefit loss against an oracle that uses true values. A second naive reference is random routing at the same clinician budget.
The strongest existing method is uncertainty-only routing: the same calibrated ensemble from Cost-Effective Hallucination Detection for LLMs, with claims sent to review in order of $p_{ij}$. It is the strongest competitor because it is what a careful engineer would build today. Its published TriviaQA F1 of 0.9106, against 0.8614 for SelfCheckGPT, shows it is a capable detector. It shares every component of our method except the consequence term, so it runs on identical extractions, identical detector features and calibration data, and the same review budget $T$. We count it as beaten if our policy recovers significantly more of the oracle's net monetary benefit at matched budgets. We pre-specify a minimum gap of half the absolute loss reduction uncertainty-routing achieves, measured at the budget where uncertainty-routing reaches half its maximum recovery. No reported result was found on Lacuna for any decision-weighted hallucination metric on these tasks. The nearest anchors are extraction F1 for Self-Verification with GPT-4 on MIMIC-IV ICD-10 coding, which rises from 0.487 to 0.533, and the PubMedBERT entity F1 of 0.69 from EvidenceOutcomes.
A deterministic margin rule flags any extracted value within a fixed distance of a guideline cutoff, for example eGFR near a KDIGO boundary or PD-L1 near 50%. It gets the same budget, with its margin width tuned on the same development split.
Two ablations isolate the ingredients. Consequence-only routing ranks by $|\Delta\text{NMB}|$ assuming constant error probability. Our full method is also run without the ungrounded-overlap feature, which tests whether the HDI-derived signal adds anything to the cheap ensemble.
Why not just…?
Why not just verify everything with Self-Verification or a sampling ensemble, since compute is cheap? We agree that compute is cheap, and we do run Self-Verification on every record. The problem is what remains afterward. Self-Verification lifts GPT-4 only to 0.533 F1 on MIMIC-IV ICD-10 extraction (Self-Verification Improves Few-Shot Clinical Information Extraction), and automatic checks cannot settle facts the model has quietly filled from priors, which is exactly the failure the Hallucination Dependence Index exposes. The "no routing" arm measures how much decision value this leaves on the table. If that residual loss is negligible relative to $\lambda$, the proposal is unnecessary, and we would report that.
Why not just route by hallucination probability? This is the main threat. The two policies agree when consequence is roughly uniform across claims. They diverge when a likely-wrong fact is decision-irrelevant, such as a comorbidity outside the eligibility rule, or when an unlikely-wrong fact sits on a cutoff for a $100,000 therapy. The matched-budget comparison settles it. We also report how the gap scales with the fraction of claims that are decision-relevant, so a null result is still informative.
Why not just flag values near the guideline cutoff? This rule is cheap, transparent, and much like the interpretable partial substitutes of Gaining Free or Low-Cost Interpretability with Interpretable Partial Substitute, which handle the easy majority of cases with rules. It is a genuine competitor for single-cutoff decisions like PD-L1 ≥50%, and our criterion contains it as a special case. It ignores two things: that error probability varies across claims (a cleanly quoted value near the cutoff needs no review), and that multi-input decisions such as KDIGO's two-dimensional eGFR-by-albuminuria grid can have consequences that are not local to one cutoff. If the margin rule matches our policy on both tasks, the right conclusion is to deploy the rule, and we would say so.
Why not just use a fine-tuned discriminative extractor that cannot invent text? Such a model can still mislabel spans; PubMedBERT reaches 0.69 entity F1 on EvidenceOutcomes. It also needs heavy annotation. The routing criterion is agnostic to the extractor, so we run it on a fine-tuned span tagger as well. The question we test is whether consequence-weighted routing helps regardless of where errors come from.
Proposed Experiments
The kidney task uses MIMIC-IV discharge summaries, the same corpus used by Self-Verification. Free-text mentions of eGFR, albuminuria, and diabetes status are extracted and mapped to KDIGO risk categories. Ground truth comes from the structured lab tables, so every extraction error is labeled automatically. Errors propagate through the re-implemented AIKD tree-plus-Markov model.
The oncology task uses the 119 TCGA-LUAD multimodal bundles from the Hallucination Dependence Index study, together with its grounded and ungrounded protocol. Stage and pathology claims feed a simplified eligibility model built on the pembrolizumab parameters. PD-L1 is not documented in those bundles, so the oncology arm tests stage-driven decisions. Biomarker-cutoff behavior is probed separately on numeric statements from NLI4CT (Saama Technologies at SemEval-2024 Task 2), where that paper's error analysis documents numerical hallucination.
The extractors are GPT-4-class and open-weight models. The primary metric is the fraction of oracle net monetary benefit recovered as a function of review budget $T$, at $\lambda$ = $100,000, $150,000, and $200,000. Secondary metrics are eligibility misclassification rate and how often the program-level ICER crosses $\lambda$ under probabilistic sensitivity analysis.
Three checks are decisive. First, at matched budgets, consequence-weighted routing must beat uncertainty-only routing and the margin rule on both tasks, with bootstrap confidence intervals over patients. Second, changing $\lambda$ must change which claims are routed. Otherwise the threshold contributes nothing beyond a constant. Third, in a study of roughly a dozen clinicians reviewing routed claims with evidence spans, we measure $q_{ij}$ and $t_{ij}$ directly. The policy's advantage must survive once simulated values are replaced with measured ones. We also test whether lower flag volume preserves the trust that AICare found fragile.
References
- Cost-Effectiveness Analysis of Artificial Intelligence-Driven Risk Stratification in Patients With Diabetic Kidney Disease in the US Veterans Population
- Cost-Effective Hallucination Detection for LLMs
- CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelines
- Self-verification improves few-shot clinical information extraction
- The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
- Cost Effectiveness of Pembrolizumab vs. Standard-of-Care Chemotherapy as First-Line Treatment for Metastatic NSCLC that Expresses High Levels of PD-L1 in the United States
- Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics
- EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes
- Gaining Free or Low-Cost Interpretability with Interpretable Partial Substitute
- Saama Technologies at SemEval-2024 Task 2: Three-module System for NLI4CT Enhanced by LLM-generated Intermediate Labels