Evidence-Conditioned Surprisal for Catching Flipped Sentences in Medical Text

A cheap way to spread health misinformation is to take a real medical article and have a language model quietly reverse one sentence: "reduces" becomes "increases," or "not recommended" becomes "recommended." The rest of the document stays authentic. Tools that detect machine-written text are poorly suited to this. They average statistical signals over the whole document, so a single rewritten sentence barely moves the score. They also cannot tell a correct sentence written by a machine from a false one. Fact-checkers have the opposite weakness: they judge truth, but they need trained veracity classifiers and ignore how a sentence came to exist. This proposal asks one question. When a biomedical language model reads a sentence, how much more surprised does it become once it is also shown the relevant medical evidence?

The core idea fits in two sentences. A flipped sentence is fluent, so a language model finds it unsurprising on its own. It becomes surprising once the model also sees evidence that contradicts it, and the size of that jump is a label-free, token-level signal that points to the exact span that was inverted. Pairing this "evidence shift" with an ordinary machine-text score gives a two-dimensional triage signal. It separates accurate automated health messaging from synthetic falsehoods, a distinction that current detectors collapse.

Core Claim

Consider a document with sentences $s_i$. Let $c_i$ be the sentences before $s_i$, and let $e_i$ be evidence retrieved for $s_i$. The proposed score is the per-token drop in log-likelihood when the evidence is added:

$$\Delta_i = \frac{1}{|s_i|}\sum_{t} \Big[\log p_\theta(x_t \mid c_i, x_{<t}) - \log p_\theta(x_t \mid e_i, c_i, x_{<t})\Big]$$

Here $x_t$ are the tokens of $s_i$ and $p_\theta$ is a biomedical sequence-to-sequence model. The encoder reads the context, and optionally the evidence. The decoder scores the sentence. A large positive $\Delta_i$ means the evidence makes the sentence less plausible, which is what a contradiction looks like. The individual terms of the sum show which tokens carry the contradiction.

The claim has three parts. First, $\Delta_i$ localizes LLM-inverted sentences inside authentic medical documents far better than document-level detectors do. Second, $\Delta_i$ stays near zero for accurate machine paraphrases, so it does not penalize machine origin as such. Third, domain pre-training matters for this signal, because the flipped tokens are clinical relation words whose likelihood depends on medical knowledge.

Research Program

Scoring model. The scorer is BioBART, which continued pre-training BART on 41GB of PubMed abstracts using text infilling. BioBART supplies the method. Its authors also found that sentence permutation hurts biomedical representation learning because discourse order carries meaning. We read that finding as a reason to condition on the preceding context $c_i$ rather than scoring sentences in isolation. The same paper documents a drop in performance from PubMed prose to colloquial dialogue. That finding constrains us: we must test on social-media-register text, not only on article prose.

Evidence sources. Polyglot persistence is the practice of storing each kind of data in the engine built for it. Here it enters as a controlled experimental variable, not as a systems contribution. We build four interchangeable evidence sources, each keyed by claim. The first is a graph store of UMLS relations, reached by linking entities to concept identifiers as in FAITH; FAITH checks decomposed triplets by graph traversal alone. The second is a full-text lexical index over HealthFC's trial and systematic-review evidence. The third is a dense vector index over the same text. The fourth is a hybrid router that queries all three, following the pattern of HetaRAG, which has only been tested on enterprise and trade documents.

The research question for this layer is specific: which kind of store supplies the conditioning text $e_i$ that exposes inversions best? We expect graph relations to do well on established drug–condition facts and full text to do well on trial outcomes. If the hybrid router adds nothing over a single hybrid lexical-plus-dense index, we will report that and drop the polyglot framing.

Evaluation set. We need hard negatives, so we build a sentence-labelled test set from Med-MMHL. That dataset already contains ChatGPT-3.5 sentence inversions inside factual medical articles across 15 conditions. We add three conditions it lacks: accurate LLM paraphrases of true sentences, inversions produced by a different generator, and inversions rewritten into calm, authoritative prose using the controllable-misinformation procedure of Can LLM-Generated Misinformation Be Detected?. That paper motivates the work: it showed this kind of style laundering lowers human and automated detection. The article-register results are then checked on the tweet portion of Med-MMHL.

Baselines

The naive reference point has two parts: random sentence ranking, and unconditional per-sentence log-rank (the $\Delta$ formula with the evidence term removed). The second tells us whether evidence adds anything at all beyond a sentence simply looking machine-like.

The document-level zero-shot detectors are log-likelihood and log-rank as benchmarked in MGTBench and DetectLLM, plus persistent-homology intrinsic dimension from Intrinsic Dimension Estimation for Robust Detection of AI-Generated Texts. That paper measures how many dimensions the text's embedding cloud occupies (about 9 for human text, 7.5 for AI text) and reports 40% accuracy at 1% false-positive rate on GPT-3.5. MGTBench reports F1 above 0.95 for log-likelihood and log-rank on its essay data, collapsing to 0.04 under perturbation. We apply these detectors in a sliding window, scoring each sentence with its neighbours, so they get a fair chance at localization.

The first supervised competitor is a BioBERT sentence classifier fine-tuned on Med-MMHL training inversions, with contrastive learning between human and synthetic variants. This recipe comes from Nuanced Multi-class Detection of Machine-Generated Scientific Text, which reports 97.60% accuracy with SciBERT on scientific abstracts. It is the strongest in-distribution competitor.

The strongest existing method for the veracity question is the evidence-based verification pipeline of HealthFC. It reports DeBERTa veracity F1 macro of 65.1% (pipeline) and 67.5% (joint), 81.9% with gold evidence, 44.4% with Google snippets, and 39.3% from claims alone. We run it on exactly the same retrieved $e_i$ given to our scorer. We also run a GPT-4-class zero-shot judge given the same evidence. For Med-MMHL sentence-level localization, no reported result was found on Lacuna, so all baselines there will be run by us.

For fairness, every method sees the same documents, the same retrieved evidence, and an equal hyperparameter-search budget on a shared development split.

Three ablations isolate the key ingredients. The first swaps BioBART for general BART, which isolates domain pre-training. The second replaces $e_i$ with a random evidence passage from the same disease, which isolates whether real evidence content matters or just extra text. The third drops $c_i$, which isolates discourse context.

Beating the field means three things. On held-out generators and style-laundered inversions, $\Delta$ must exceed the fine-tuned BioBERT classifier in sentence-level AUROC. It must match or beat the HealthFC-style verifier's F1 without using veracity labels. And it must produce significantly fewer flags on accurate machine paraphrases than any origin detector.

Why not just…?

Why not just fine-tune BioBERT to classify flipped sentences? In distribution, it will probably win, and we expect that. The weakness is transfer. Classifiers learn the traces of a particular generator, and LLMFake shows that rewriting erases surface markers. The held-out-generator and laundered conditions settle this directly. If the classifier holds up there as well, the label-free argument weakens to cost alone, and we will say so.

Why not just run an evidence-based entailment classifier like HealthFC's pipeline on each sentence? This is the real threat, because it uses the same evidence. The differences are that $\Delta$ needs no veracity-labelled training data, points to the flipped tokens, and yields an origin-independent score. None of that matters if entailment is simply more accurate. The head-to-head on identical $e_i$ decides it. If entailment matches $\Delta$ everywhere, the contribution reduces to the localization and triage analysis, and we will frame the paper that way.

Why not just use a document-level or windowed zero-shot detector? These detectors measure machine origin, not falsehood. By construction, they should flag accurate paraphrases as often as inversions. The paraphrase-negative condition measures this false-flag rate directly. MGTBench's collapse under perturbation suggests they will also fail on laundered text.

Why not just use one hybrid vector index instead of several stores? This may well be enough. HealthFC's weak result with web snippets shows that evidence quality matters, but not that multiple engines matter. The per-store ablation is designed to answer exactly this, and a null result retires the polyglot component.

Proposed Experiments and Decisive Checks

The primary experiment reports sentence-level AUROC, top-1 localization accuracy within each document, and true-positive rate at 1% false-positive rate on the extended Med-MMHL set. Results are broken down by condition: same generator, new generator, laundered, and tweet register. A second experiment uses $\Delta$ thresholded on the development split as a zero-shot veracity scorer on HealthFC claims. It is compared against the reported 65.1% and 67.5%, and against the 81.9% oracle ceiling when given gold evidence.

Three checks decide the proposal. The first is the random-evidence ablation: if shuffled evidence yields similar $\Delta$, the signal is length or topic, not contradiction, and the idea fails. The second is the 2D triage plot of unconditional log-rank against $\Delta$. It must separate accurate machine text, human truth, and machine inversions into distinct regions, or the Gap 1 claim fails. The third is the counter-evidence test motivated by Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation, which argues that direct refutations often do not exist yet for new rumors. We remove the directly relevant evidence from every store and measure how far $\Delta$ degrades. This sets an honest bound on where the method can work during a new outbreak. Finally, the BioBART versus BART comparison on tweet-register text tests whether scientific pre-training helps or hurts in the lay register where infodemics actually spread.