Many misleading health claims are not false. "Cancer drug reverses Alzheimer's memory loss" can rest on a real experiment: mice engineered to develop amyloid plaques were given a drug for five months, and they then swam to a hidden platform more efficiently in a Morris water maze. The Morris water maze is a pool of opaque water in which a rodent must find a submerged escape platform using cues on the room walls, and it is the standard behavioral test of rodent spatial memory. The experiment supports something, but much less than the headline says. The claim widens the species, the disease, the outcome measure, and the strength of the effect all at once. Current fact-checkers handle this badly. They give one SUPPORTS or REFUTES label, or one continuous score, and they cannot tell a reader which part of the claim went beyond its evidence.
We propose a verifier for preclinical-to-clinical claims that works in two steps. First it breaks each claim and each cited study into a small set of scope slots: species and disease model, intervention and dose, measured outcome, and effect direction and strength. Then it compares the slots one at a time. The final verdict is a fixed, inspectable function of those comparisons, so the explanation is the actual cause of the verdict and not a story written afterward. The water maze is where we test this, because it is common enough to supply hundreds of real studies and specific enough to support checks that generic verifiers lack. One example is whether a study reporting only escape latency can back a claim about memory rather than learning, or about cognition when the authors did not control for swim speed or wall-hugging.
Core Claim
Our claim is that overgeneralization ("specificity mismatch") is the main failure in biomedical claim verification and that it has an explicit structure. If a model represents that structure in fixed slots, comparing claim to evidence slot by slot, two things should follow. It should match a strong prompted language model on verdict accuracy. It should also clearly beat that model on faithfulness: when one slot of a claim is edited, the verdict should change in the predicted way and only then. We also expect it to report which dimension was overstated far more accurately than any free-text rationale. Faithfulness is the headline claim. Accuracy parity is the minimum requirement.
Why This Gap Is Real
The motivation comes from SciFact-Open. That paper tested verifiers against 500,000 biomedical abstracts. It found specificity mismatches in 44% of annotated evidence pairs and conflicting evidence for about 20% of well-studied claims. MultiVerS fell from 72.5 to 52.4 F1 when moved from the curated to the open setting. SciFact-Open diagnoses the problem but models it only as three-way classification. VERT5ERINI builds a strong retrieve, rerank, select-rationale, and label pipeline, but its authors say it cannot reason over effect sizes or statistics, and its explanations are extracted sentences. DiVA goes beyond binary labels with agentic search and a scalar factuality score trained by pairwise ranking. A scalar score still says how wrong a claim is, not along which axis. DiVA was also evaluated only on general question answering. FineFake has a health domain and a "content-knowledge inconsistency" label, but its knowledge consists of Wikidata triples, not experimental designs. Fine-grained taxonomies such as Fakeddit and the truthfulness, harm, and verifiability axes of Schlicht et al. label the type of misinformation. Neither breaks a scientific claim into the parts that can be overstated.
The preclinical papers in the file are the raw evidence. Lee et al. report ibrutinib at 10 mg/kg rescuing water-maze memory in 5xFAD mice. Zhao et al. report baicalin shortening escape latency (P = 0.008) and raising target-quadrant time (P = 0.015) in rats injected with Aβ1-40. Related work covers bisdemethoxycurcumin, SFPQ overexpression, resveratrol in sepsis encephalopathy, and exercise in D-galactose-aged rats. Each of these is a finding that is real but narrow, and exactly the kind that tends to be overstated.
The method is borrowed from clinical interpretability. From Self-Verification we take the steps of omission detection, verbatim evidence grounding, and pruning of unsupported extractions. In that work, grounded spans overlapped 93% with clinician-annotated spans, and F1 on clinical trial arms rose from 0.419 to 0.530. From Clinical Graph of Thoughts we take the lesson that forcing a model through clinically meaningful intermediate quantities prevents collapse on hard prognostic targets. In that paper an intermediate MRI injury score raised 2-year outcome accuracy to 71.73% from roughly 52–56%. Our intermediate quantities are scope slots rather than imaging scores.
Two findings constrain the design. The interpretability literature reports that post-hoc attributions such as SHAP can disagree across equivalent models, so we do not explain verdicts with attributions. The clinical XAI literature warns that generated rationales can be fluent and still unfaithful (Explainable AI in Clinical Practice). We therefore let the slots set the verdict.
Research Program
The system has three parts.
The first is a slot extractor. It reads a study's abstract and, where available, its methods and results text, and fills typed slots. These are: organism and disease model, for example "5xFAD transgenic mouse" or "Aβ1-40 intracerebral injection, rat". Intervention, dose, and duration. Outcome measures, using a water-maze vocabulary of escape latency, path length, probe-trial quadrant dwell, and platform crossings, plus molecular markers. Reported controls, such as swim speed, visible-platform trials, and thigmotaxis. Effect direction and statistical support. A second pass looks for omitted slots, and every filled slot must cite a verbatim span. Slots without a supporting span are pruned, following Self-Verification.
The second part fills the same slots from the claim. Unstated slots are marked "implied general", so that "reverses memory loss" with no species named is read as implying humans.
The third part is a comparator. Each slot has an ordinal scope ladder. For species, the ladder runs cell culture, then rodent, then non-human primate, then human. For outcome, it runs a single water-maze metric, then spatial learning, then spatial memory, then cognition, then clinical disease modification. The comparator decides, slot by slot, whether the claim stays within the evidence, goes beyond it by a given number of rungs, or contradicts it.
Some water-maze rules are written in explicitly. Escape latency without path-length or swim-speed control supports "learning" only weakly. Only probe-trial measures support "memory retention". An injected-Aβ model does not support claims about the progressive disease. The verdict is the worst slot outcome on an ordinal scale: supported, mildly overstated, severely overstated, contradicted, or no evidence. The overstated slots, together with their spans, form the explanation.
The ladders and rules will be written with two biomedical annotators and released. A learned component produces calibrated probabilities for each slot comparison so that borderline cases are not hidden. The final aggregation stays fixed.
The benchmark, which we call MWM-Claims, has two parts. The real part pairs about 300 preclinical water-maze papers with claims collected from press releases, news stories, and review sentences that cite them. Expert annotators label each pair with the slot-level judgments and the overall verdict. The source abstracts are drawn from or added to the S2ORC corpus used by SciFact-Open, so retrieval can be tested in the open setting. The counterfactual part takes each real claim and edits exactly one slot, either widening or narrowing it. This tells us in advance which verdict should change and in which direction. This part is used only to measure faithfulness, never to measure accuracy, because templated edits are easier than real phrasing.
Baselines
The naive reference labels every claim SUPPORTED when it shares a drug and a disease term with a retrieved abstract that reports positive results. This stands for what keyword matching does on gray-zone health claims. Any useful system must beat it clearly on overstated claims, where it should score near zero by construction.
The strongest existing scientific verifiers are MultiVerS as evaluated in SciFact-Open, reported at 52.4 F1 in the open setting, and VERT5ERINI, reported at 65.07 full-pipeline F1 on closed SciFact against 55.31 for VeriSci. We will run both on MWM-Claims using the same retrieval corpus. Their three-way output will be mapped onto our verdict scale, with overstated claims counted as correct when labeled REFUTES or NOINFO. We will also fine-tune both on the MWM-Claims training split so they see the same supervision as our system. No reported result exists on Lacuna for any system on preclinical-to-clinical overgeneralization, so these numbers anchor what is plausible but cannot be compared directly.
The strongest likely competitor is a frontier language model. It will be given the same retrieved evidence and the same slot definitions in its prompt, and asked for an ordinal verdict plus a rationale naming the overstated dimension. Its prompting budget will be equal to our system's total number of calls, including several samples with majority vote. A DiVA-style variant (DiVA) with agentic search and a scalar regression head will also be trained on our split. DiVA reports 84.24% precision@1 on Natural Questions against 77.64% for FactScore, and 79.87% on Musique against 57.14% for a naive generative verifier. These results show it is a strong ranker, though on other domains.
Three ablations isolate the key ingredients. The first keeps slot extraction but lets a language model produce the verdict freely from the filled slots, which tests whether fixed aggregation matters. The second removes the water-maze-specific rules and ladders, which tests whether domain structure matters beyond generic scope slots. The third removes evidence grounding and pruning. To count as beating the frontier model, our system must be within 2 macro-F1 points of it on real claims. It must also be significantly better, by a paired bootstrap test, on counterfactual consistency and on identifying which slot is overstated.
Why not just…?
Why not just prompt a strong language model to give a nuanced verdict with a rationale? This is the real threat, and it may well match us on verdict accuracy. Our claim does not depend on beating it there. What the prompted model cannot guarantee is that its rationale caused its verdict. The clinical XAI literature in the file names fluent but unfaithful rationales as an open problem. The counterfactual split tests this directly. If the prompted model's verdicts track single-slot edits as reliably as ours, then a fixed slot structure adds nothing and the proposal should be reduced to a benchmark contribution. We will report that outcome if it occurs.
Why not just fine-tune a SciFact-style verifier with an extra "overgeneralized" label? That gives a flag without saying which part of the claim is wrong. Readers and fact-checkers need to know which part, as Nakov et al. and MisinfoEval argue on the grounds that corrections without specific explanation are not taken in. The fine-tuned MultiVerS and VERT5ERINI baselines test whether the extra label alone recovers verdict accuracy. Slot identification would remain impossible for them.
Why not just use a continuous factuality score, as DiVA does? A score of 0.4 does not separate "right drug, wrong species" from "right species, latency mistaken for memory". Those two errors call for different corrections. We will measure whether DiVA's scores rank claims correctly within the counterfactual families. If they do, the score could be combined with our slots rather than replaced.
Proposed Experiments
The first decisive check is counterfactual consistency. For each one-slot edit, we record whether the verdict changes in the predicted direction and stays the same for edits that should not matter, such as paraphrasing an unchanged slot. We expect our system to approach full consistency by construction, up to extraction errors. We expect the prompted model and fine-tuned verifiers to fall noticeably short. Failing to beat them here would refute the core claim.
The second check is accuracy on real claims: macro-F1 over the ordinal verdicts and accuracy at identifying the overstated slot. It is reported separately for claims about the water-maze outcome measure, where the domain rules should help most, and for claims about species or model.
The third check is transfer. We run the generic slot comparator, without water-maze rules, on SciFact-Open claims annotated as specificity mismatches. This tests whether the structure helps beyond the maze, measured against the reported 52.4 F1 of MultiVerS.
The fourth check is a small user study modeled on the clinician studies in TxGNN and Jungmann et al.. Biomedical readers judge claims with no aid, with the prompted model's rationale, or with our slot explanation. We measure accuracy at finding the overstatement and time taken.
If our system wins on consistency and slot identification, keeps accuracy parity, and helps readers find the overstatement, the evidence supports the claim that structured scope comparison is the right way to verify gray-zone biomedical claims.
References
- SciFact-Open: Towards open-domain scientific claim verification
- Scientific Claim Verification with VerT5erini
- DiVA: Fine-grained Factuality Verification with Agentic-Discriminative Verifier
- FineFake: A Knowledge-Enriched Dataset for Fine-Grained Multi-Domain Fake News Detection
- Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection
- Co-Creation of Misinformation Management Policies
- Chronic oral administration of ibrutinib prevents long-term memory deficits and reduces AD pathology and neuroinflammatory responses in a mouse model of AD
- An interventional study of baicalin on neuronal pentraxin-1, neuronal pentraxin-2, and C-reactive protein in Alzheimer’s disease rat model
- Bisdemethoxycurcumin inhibits oxidative stress and antagonizes Alzheimer's disease by up‐regulating SIRT1
- Overexpression of SFPQ Improves Cognition and Memory in AD Mice
- Resveratrol Protects against Sepsis-Associated Encephalopathy and Inhibits the NLRP3/IL-1βAxis in Microglia
- Moderate-Intensity Intermittent Training Alters the DNA Methylation Pattern of PDE4D Gene in Hippocampus to Improve the Ability of Spatial Learning and Memory in Aging Rats Reduced by D-Galactose
- Self-verification improves few-shot clinical information extraction
- Visual and Domain Knowledge for Professional-level Graph-of-Thought Medical Reasoning
- Perspective of Artificial Intelligence in Disease Diagnosis: A Review of Current and Future Endeavours in the Medical Field
- Automated Fact-Checking for Assisting Human Fact-Checkers
- MisinfoEval: Generative AI in the Era of "Alternative Facts"
- A foundation model for clinician-centered drug repurposing
- Algorithmic transparency and interpretability measures improve radiologists’ performance in BI-RADS 4 classification