Gate Before Verdict: Evidence-Gated Verification for Clinical Claims

Language models that check medical claims often answer "true" or "false" when the evidence in front of them does not support either verdict. Sometimes they read cues in the claim's wording instead of the evidence. Sometimes they turn a tentative finding into a firm one. Sometimes the verdict depends on a background fact that nobody wrote down. Sentiment analysis faced a similar problem decades ago. Flat classifiers trained on mostly neutral text kept confusing "says nothing" with "says something negative." The fix was a two-stage cascade: first decide whether the text expresses an opinion at all, then decide its direction. We propose the same split for claim verification. A gate first asks whether the evidence commits to any verdict. A second stage decides entailment or contradiction only for claims that pass.

The gate is trained in a way that a flat classifier cannot easily copy. Every training claim is shown both with its evidence and with the evidence removed or scrambled, and the gate is taught that evidence-free inputs must be judged "insufficient." The gate can also pass a claim along a third route, "needs an unstated premise," which triggers a domain-tuned generator to propose the missing background fact. That generator borrows from induction-augmented generation, a method that writes out commonsense facts a retriever cannot find. The main bet is that a verifier whose first decision is forced to depend on the evidence will stop exploiting surface cues in the claim. We test this on clinical-trial inference and biomedical question answering. On those benchmarks, a model that never sees the evidence already scores nearly as well as specialized biomedical models that do.

Core Claim

On NLI4CT, the clinical-trial inference benchmark of NLI4CT, a BERT model trained on the statements alone reaches F1 0.626. BioBERT with full access to the trial reports reaches 0.644. Almost all of the measured skill can therefore be produced without reading the evidence, which is a hallucinated form of grounding. Our claim is that factoring the decision as

$$ p(y \mid s, e) = p(g = \text{commit} \mid s, e)\; p(y \mid s, e, g = \text{commit}) $$

does something a flat softmax does not, when the gate is trained with the evidence-ablation constraint described below. Here $s$ is the claim, $e$ the evidence, $g$ the gate's decision (commit, needs-premise, or insufficient), and $y$ the verdict. The constraint pushes $p(g=\text{commit} \mid s, \tilde e)$ toward zero whenever $\tilde e$ is empty, belongs to another trial, or has its key numbers corrupted.

The gate is soft. Its probability multiplies the verdict probability during joint training, which avoids the problem of an early stage throwing away evidence for good. Strict two-stage pipelines such as Subjectivity and Sentiment Analysis of Modern Standard Arabic and The Cool Cucumber System suffer from exactly that problem. If the claim is right, three things should happen together. Statement-only accuracy should collapse toward chance. Consistency under meaning-preserving rewrites of claims should rise. Full-evidence F1 should hold or improve. A flat model given the same augmented data should not match all three.

What This Builds On and Where It Departs

The gate idea comes from hierarchical sentiment work. That work showed that separating "is there an opinion" from "which way" helps when the data are dominated by neutral text. One example is the two-stage SVM pipeline of Abdul-Mageed et al., with a subjectivity stage at F 71.54% feeding a polarity stage. Another is Pang and Lee, who showed that how classes relate to one another should shape the decision. We move the neutral-versus-polar split from opinion to evidential commitment. PubMedQA's yes/no/maybe labels are a direct analogue of objective/positive/negative.

The motivation comes from two lines of hallucination work. Tang et al. found that models summarizing clinical evidence turn conditional findings into unconditional facts. NLI4CT exposed the statement-only bias.

The method for the needs-premise route comes from IAG. IAG uses a prompted generator to produce class-level background facts and lets a fusion reader filter out bad ones. IAG was tested only on general yes/no commonsense questions, with generic T5 and GPT-3 models. We apply it to clinical premises such as dose ordering or eligibility logic, and we let the gate decide when induction is needed at all.

Shwartz and Choi supply a scientific question that our design can answer. They showed that language models amplify reporting bias, the tendency of text to mention the unusual and omit the obvious. They left open whether pretraining on domain text reduces or worsens this bias. Comparing a general premise generator with a domain-adapted one, and scoring both against gold premises, gives a measurement in the clinical domain.

The closest existing system is Saama's three-module NLI4CT pipeline. It generates diagnostic questions, answers them against the report, and aggregates. Its modules are hard-chained, it has no stage that is trained to depend on the evidence, and its best submission reached F1 0.659. We differ in three ways: the routing is soft, the gate's objective is defined by evidence ablation, and premise induction runs only when the gate asks for it.

Edinburgh's adapter merging shows that a contrastive triplet adapter improves consistency by 0.152. That result partly overlaps with our goal. We treat it as the strongest open-model competitor rather than as prior art for the gate.

Two findings limit the design. NLI4CT's ceiling at 0.644 F1 means domain pretraining alone cannot be the answer, so it appears here only as an input to the premise generator and the verdict stage. Edinburgh's observation that chain-of-thought prompting lowered raw F1 argues against making the gate a free-text reasoning step. The gate is a classification head, and only the premise route produces text.

Research Program

The base model is a 7B open model, MistralLite as in the Edinburgh system, so that compute is comparable. It is used in two variants. The general variant is used as is. The domain variant adds a language-modeling LoRA adapter trained on PubMed abstracts, following Edinburgh's language-modeling adapter. Following BioBART, we use no sentence-shuffling objective, because that work found shuffling hurts biomedical discourse.

The gate and the verdict stage are two heads on a shared encoder and are trained jointly with the factored likelihood above. The gate receives extra supervision from three sources. The first is evidence-ablated copies of every claim. The second is copies where the evidence comes from a different trial, the wrong report section, or has its numbers altered. The third, on PubMedQA, is the gold "maybe" label, with the 211.3k automatically generated PQA-A instances used for pretraining the gate.

On the needs-premise route, a premise generator writes several candidate background facts. There are two versions: a prompted LLM and a distilled student in the style of IAG-Student, each with general and domain-adapted versions. The candidates are appended to the evidence, and the gate is run again. A premise is kept only if it moves the gate toward commit and survives a check for agreement across samples. This is the IAG filtering idea used as a gate signal rather than inside a fusion reader.

For analysis, we annotate a 300-item NLI4CT sample with the gold unstated premise, where one exists. This lets us measure how often each premise generator produces correct, plausible-but-wrong, or prototype-biased premises. Prototype bias here is the clinical analogue of Shwartz and Choi's "bananas are red" errors.

Baselines

The naive reference points are the majority class, the statement-only BERT at F1 0.626 reported in NLI4CT, and on PubMedQA the majority guess at 55.2% accuracy reported in PubMedQA. Any system that does not clearly exceed the statement-only figure is not reading the evidence.

The strongest closed reference is zero-shot GPT-4 on NLI4CT. The file reports an average score of 0.8328 and faithfulness of 0.9479 (Edinburgh). We report it as a ceiling and do not claim to beat it with a 7B model. If API access allows, we also run GPT-4 with the gate posed as a first question, to see whether the factoring helps even a strong prompted model.

The strongest open competitor is Edinburgh's merged language-modeling plus triplet LoRA adapter on MistralLite-7B, which reports an average score of 0.7862. We retrain it on our base model with the same training data, including our ablated copies used as extra contradiction-style negatives, the same number of GPU hours, and the same hyperparameter-search budget. Beating it means a higher official average score together with a statement-only score at least 10 F1 points lower than the adapter's.

The pipeline competitor is Saama's three-module system, with best reported F1 0.666 and best consistency 0.657 across its own ablations. We reimplement it on the same base model so that the comparison isolates soft gating versus hard question-and-answer chaining. On PubMedQA the reference is multi-phase BioBERT at 68.1% accuracy, against a human score of 78.0%.

The ablation that matters most is a flat three-way classifier (entail, contradict, insufficient) trained on exactly the same evidence-ablated data. It has the same parameters and budget but no factorization and no separate gate head. Further ablations are hard routing versus soft routing, removing the premise route, and general versus domain premise generators.

Why not just…?

Why not just train a flat classifier with the ablated examples added as a third class? This is the most serious threat. A softmax over three classes can represent our factored distribution, so any gain has to come from the separate gate loss and from the gate's role in premise induction, not from the hypothesis class. We expect the flat model to learn "insufficient when evidence is empty" as a surface rule, while still using claim wording on real evidence. Whether that happens is an empirical question. The decisive test uses evidence that is present but irrelevant, such as another trial's report or altered numbers. Those cases are held out from training. Testing on them separates real evidence dependence from detecting empty input. If the flat model matches the gated model there, we drop the hierarchy claim and report the augmentation as the contribution.

Why not just apply conformal prediction or confidence-based abstention to a strong classifier? TRAC reaches 0.90 empirical coverage at α=0.1 on Natural Questions. However, its guarantee bounds how often the true answer is missing from the candidate set. It does not check whether the confidence was earned from the evidence. A classifier that relies on statement cues can be confidently and consistently wrong, and conformal sets built on its scores will be tight and wrong-headed. We test this by comparing confidence on statement-only inputs: a calibrated flat model should stay confident there, and the gate should not. Conformal calibration is compatible with our gate, and we apply it on top as a final layer.

Why not just use domain pretraining, or GPT-4 with chain-of-thought? NLI4CT reports BioBERT and BioMegatron stuck at 0.644, which is barely above statement-only. Edinburgh found that chain-of-thought prompting lowered F1. GPT-4 is strong but closed and cannot be retrained with ablation constraints. Domain adaptation stays in our system as an input to the other components, and the domain-versus-general premise generator comparison measures what it contributes.

Why not just run IAG directly? IAG was built for general yes/no commonsense questions with generic models, and it induces premises for every question. Our premise-route ablation tests whether gate-triggered induction beats inducing on every item. Induction on every item adds premises to items that do not need them, which is where plausible-sounding but wrong generalizations can enter.

Decisive Checks

We evaluate on the NLI4CT test set using the official F1, faithfulness, and consistency metrics with the EdinburghClinicalNLP/semeval_nli4ct code, and on PubMedQA PQA-L using accuracy and macro-F1. We also report three diagnostics: the statement-only score, the score with irrelevant evidence, and the overcommitment rate on PubMedQA, defined as the fraction of gold "maybe" items given a yes or no answer, which is the direct test of the overconfidence Tang et al. describe.

The proposal succeeds if two things hold. First, against the flat-with-augmentation ablation, the gated model lowers statement-only and irrelevant-evidence performance to near chance while matching or exceeding full-evidence F1. Second, it exceeds Edinburgh's 0.7862 under equal compute.

The premise study succeeds if domain-adapted generators produce measurably fewer prototype-biased premises than general ones. A null result would also be informative: it would show that reporting bias carries over into domain corpora, which addresses the question Shwartz and Choi left open for clinical text.