Misleading health claims often cite a real paper correctly. The claim repeats a true sentence from the results ("compound X reduced tumor growth"), and that sentence is easy to find because it shares the claim's words. What makes the claim false sits elsewhere: the methods say the study used mice, or the limitations say the design was observational. Those sentences share almost no words with the claim, so retrievers miss them. A language model that sees only the supporting sentence then endorses the misinformation. This proposal attacks that failure directly. Before retrieving, we turn each claim into a short set of questions about what it silently assumes, such as which organism was studied, what the study design was, and what dose or outcome was measured. Each question is then answered from the part of the paper whose job is to state that fact, so methods questions go to methods sentences and caveat questions go to limitation sentences.
The bet is that the undermining passage is lexically far from the claim but lexically close to a predictable question about the claim. "In mice" does not match "cures cancer," but it does match "what organism was studied?" Reading only the routed sentences is also much cheaper than having a large model read every full paper with quadratic-cost attention. That cost is the binding constraint once the verifier has to consider many candidate papers rather than one.
Core Claim
Grounding Fallacies Misrepresenting Scientific Publications in Evidence introduces MISCIPLUS, 114 fallacious arguments grounded in 100 biomedical papers. It shows that retrievers find the "kernel of truth" passage but not the undermining context. Giving Llama 3 the kernel passage raised its acceptance of misinformation from 0.0% to 44.4%. That paper diagnoses the problem but does not propose a retrieval mechanism for the undermining context. We claim that a fixed, small set of assumption types (organism or population, study design, exposure or dose, outcome measured, effect size, stated limitations) is enough to recover most undermining contexts when each type is routed to its matching rhetorical role in the full text. We further claim this cuts fallacy acceptance well below what either kernel-only retrieval or claim-paraphrasing retrieval achieves. Our approach differs from query rewriting such as RaFe (Mao et al.) and InteR (Feng et al.), which rephrase the claim itself. We instead generate questions the claim does not state and restrict where in the document each one may be answered.
Research Program
Three components are built.
Claim-to-questions decomposition. An instruction-tuned model maps each claim into slot-specific questions drawn from the fixed schema, and it is told to state the claim's implicit value for each slot (for example, "implies humans"). This resembles the atomic decomposition of NSnet (Bridging Knowledge Gaps in Neural Entailment via Symbolic Models), which split hypotheses into subject-verb-object tuples and matched them against a knowledge base, reaching 77.9% on SciTail. The difference is in what gets decomposed: we target the assumptions a claim leaves unsaid rather than its explicit content, and we match against the source paper rather than an external knowledge base.
Full-text rhetorical role tagger. A sentence-level model labels every sentence of a paper as background, objective, method, result, conclusion, or limitation. It is run linearly over sentences, so its cost grows with paper length rather than with its square. It starts from the sequential labeling setup of PubMed 200k RCT, where a bidirectional recurrent model with a label-sequence layer reached 91.6% F1 on abstracts. We add a limitation label and adapt the tagger to full text using section headers as weak labels.
This is where semantic category overlap enters as a measured risk rather than a solved problem. That paper found background and objective sentences are the main source of confusion, and Nigam et al. found the same blur between facts and reasoning in legal text. In our setting the dangerous confusion is between hedged result sentences and limitation sentences. A routing error there sends a caveat question to the wrong pool. We will measure how much each confusion type costs downstream, not just tagger F1.
Routed retrieval and verification. Each question is scored only against sentences carrying its allowed roles, with a cross-encoder. The top sentences per slot are passed, together with the kernel passage, to the verifying model. The verifier must fill each slot with the paper's value and compare it with the claim's implied value before giving a verdict. The model reads only a few dozen sentences per paper.
Baselines
The naive reference point is the condition already reported in Grounding Fallacies: the verifier sees only the kernel passage. That condition is reproduced with our verifier models, and the published Llama 3 acceptance rate of 44.4% serves as an anchor. The paper also reports kernel retrieval of up to 0.664 MRR (mean reciprocal rank, where 1.0 means the correct passage is always ranked first). We found no reported result on Lacuna for retrieval of the undermining context on MISCIPLUS, so we will establish that number ourselves with BM25 and a dense retriever.
The strongest existing competitor is a long-context model that reads the whole paper. This attacks the problem by removing retrieval altogether. We will run the same verifier families on full text truncated only by their context limits, using the same prompts and the same verdict format. We will record total tokens processed and wall-clock time alongside accuracy. Beating it means matching or lowering its fallacy acceptance rate while reading roughly an order of magnitude fewer tokens. Clearly lowering its acceptance rate would show that routing helps accuracy and not only cost.
A second strong competitor is claim rewriting in the RaFe and InteR style. Here a model rewrites the claim several times, or is asked "what evidence would undermine this?", followed by dense retrieval over all sentences. It receives the same number of model calls and retrieved sentences as our method.
For document-level verification, MultiVerS (MultiVerS) jointly predicts verdicts and rationales over whole abstracts with Longformer. It is run on the SciFact-style experiments with its released configuration.
Two ablations isolate the key ingredient. The first keeps the questions but removes role routing, so questions search all sentences. The second keeps routing but replaces typed questions with the raw claim. A third ablation replaces the learned tagger with section-header rules.
Why not just…?
Why not just give the whole paper to a long-context model? This is the real threat. At 5,000 to 15,000 tokens per paper, modern models can read a single paper. "Evidence overruling knowledge" (Grounding Fallacies) suggests that a prominent supporting passage can dominate the verdict even when caveats are present. And HaluEval reports ChatGPT detecting summarization hallucinations at only 58.53% accuracy, which hints that reading more text does not by itself produce checking. Neither result proves full-text reading fails, though. The full-paper baseline will settle it. If it matches our accuracy, we will report that the contribution is cost and auditability, not accuracy. Cost matters in the open-domain setting, where the verifier would otherwise read every candidate paper in full.
Why not just use section headers instead of a learned role tagger? Many papers have a "Limitations" heading, and header rules may be enough. Headers fail when caveats are scattered through discussion paragraphs or when abstracts and preprints lack structure. That gap is exactly where the role overlap documented in PubMed 200k RCT applies. The header-rule ablation tests this directly. If it matches the learned tagger, we drop the tagger and the contribution rests on the typed decomposition.
Why not just rewrite the query to ask for counter-evidence? Generic counter-evidence prompts still produce text in the claim's vocabulary. SciFact-Open found that dense retrieval is distracted by topically related but specificity-mismatched evidence, with a mismatch in specificity in 44% of annotated evidence pairs. The matched-budget rewriting baseline, together with the lexical-overlap analysis below, will show whether typed questions add anything beyond a good counter-evidence prompt.
Proposed Experiments
The primary benchmark is MISCIPLUS with full texts. The main metrics are recall@k and MRR for undermining-context passages, and fallacy acceptance rate per verifier. With only 114 arguments, all comparisons use paired bootstrap confidence intervals.
The mechanism check is a lexical-overlap measurement. For each gold undermining passage, we compute content-word overlap with the claim and with the routed question for the relevant slot. The hypothesis predicts near-zero overlap with the claim, consistent with Yang et al., and clearly higher overlap with the question. If question overlap is no higher, the vocabulary-bridging story is wrong even if accuracy improves.
Generalization is tested on SciFact, HealthVer and COVIDFact, using abstract-level verdict F1 against MultiVerS. We also test the open setting of SciFact-Open, where MultiVerS fell from 72.5 to 52.4 F1. There we ask whether slot checks reduce acceptance of topically related but mismatched abstracts, at fixed token budget.
The tagger is evaluated on PubMed 20k and 200k RCT, against the 90.0% and 91.6% F1 reported for those sets, and on a small hand-labeled full-text set with limitation sentences. We then inject controlled confusions, swapping result and limitation labels at increasing rates, to trace how routing errors raise acceptance.
The decisive outcomes are as follows. Routed retrieval should raise undermining-context recall over both rewriting and unrouted questions. It should cut acceptance below the kernel-only condition and at least match full-paper reading at a fraction of the tokens. If unrouted questions do as well as routed ones, the rhetorical-role component is unnecessary. If full-paper reading wins on accuracy, the proposal reduces to an efficiency method, and we will say so.
References
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- Bridging Knowledge Gaps in Neural Entailment via Symbolic Models
- PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstracts
- LegalSeg: Unlocking the Structure of Indian Legal Judgments Through Rhetorical Role Classification
- MultiVerS: Improving scientific claim verification with weak supervision and full-document context
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- SciFact-Open: Towards open-domain scientific claim verification
- WikiQA: A Challenge Dataset for Open-Domain Question Answering