Knowing the Fact vs. Binding It: Minimal Pairs for Biomedical Hallucination

When a language model writes that a drug "reduced mortality in the placebo arm" or credits one trial population with another's outcome, two different failures can produce the same wrong sentence. The model may not know the medical fact. Or it may know the fact and still attach it to the wrong entity in the text in front of it. Benchmarks that grade whole answers cannot tell these apart, and domain-specific pre-training is mostly justified as a fix for the first failure.

We propose to separate the two failures with minimal pairs: pairs of sentences that are identical except for one controlled change, where we check which version the model assigns higher probability. The pairs are built from biomedical statements. Each knowledge variant is crossed with a variant where distracting entities sit between the entity and its attribute. Running these pairs on models whose only difference is pre-training data (biomedical text from scratch, general web text, or a mix) tests a specific bet. Domain pre-training should improve stored-fact preference but leave the "attach it to the nearest noun" failure largely intact. That binding failure, not missing knowledge, should predict the entity–attribute swaps that clinical summarization studies report.

Core Claim

Three findings in the literature have not been put together. COMPS built minimal pairs over everyday concept–property facts ("a robin can fly" vs. "a penguin can fly") and found a proximity effect. Models average about 68% on plain property attribution but fall to 47.2% when a distractor concept is placed between premise and property, and larger models fall further below chance. Evaluating large language models on medical evidence summarization found two things in clinical summaries: errors that scramble Patient/Intervention/Comparison/Outcome elements, and a "More is Less" effect where longer source context increases factual errors. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing showed that pre-training from scratch on PubMed with an in-domain vocabulary beats general and mixed-domain models (BLURB 81.16) and that general text causes negative transfer.

Our hypothesis is that the clinical attribute-swap failure is the COMPS proximity effect operating on biomedical content. If so, domain pre-training should raise accuracy on undistracted factual pairs while the accuracy drop caused by in-between distractors stays roughly constant across pre-training regimes. The claim is falsifiable both ways. If domain-pretrained models also show a much smaller distractor drop, then domain data repairs binding, and the case for scratch pre-training as a hallucination remedy gets much stronger.

Research Program

The benchmark

The benchmark, which we call BioCOMPS, translates the COMPS design into a 2×2 factorial over biomedical assertions. The design borrows the crossing logic used for filler–gap dependencies in What do RNN Language Models Learn about Filler-Gap Dependencies?.

The first factor is the knowledge source.

  • Real-entity items test stored knowledge: a real drug, its real mechanism class, and a real adverse effect.
  • Novel-entity items introduce an invented entity through an in-context premise ("Zorvatide is a beta blocker. Therefore, Zorvatide can cause bradycardia"). This mirrors the COMPS made-up-word ("wug") construction. Stored knowledge about the entity is impossible, so the item measures binding plus class knowledge.

The second factor is distractor placement: none, before the premise, or between premise and attribute. Each distractor is a second drug or patient group whose class licenses a different attribute.

Item sources:

  • Drug class and adverse-effect pairings.
  • Trial-arm statements templated from the PICO error categories in the Cochrane summarization study, such as which arm improved and which population was enrolled.
  • Case-report findings drawn from PMC-Patients.

Every item is checked by clinicians, following the expert-validation practice of MHB.

Confound controls

The foil differs from the correct sentence only in which entity heads the attribute. Wherever possible we score the surprisal (negative log-probability) of the attribute tokens given an otherwise identical prefix, rather than the whole-sentence probability. Each attribute appears as correct in some items and as the foil in others, so attribute frequency cancels out. Pairs are matched on token count under each tokenizer, because Token-length Bias in Minimal-pair Paradigm Datasets shows that length differences alone can decide minimal-pair outcomes.

Probability differences between sentences mix plausibility with form (What Can String Probability Tell Us About Grammaticality?). Here the form is held fixed, so the remaining difference is the factual binding we want to measure.

The models

The causal lever is pre-training data, so the core comparison uses decoder models we train ourselves at two sizes (roughly 125M and 350M parameters) with matched token budgets. There are four regimes:

  1. PubMed abstracts from scratch with a PubMed-trained tokenizer. This is the corpus scale used for PubMedBERT, about 3.2B words.
  2. The same PubMed data with a general-English tokenizer.
  3. General web text only.
  4. General pre-training followed by continued pre-training on PubMed.

Regimes 1 and 2 isolate the vocabulary effect that PubMedBERT credits for part of its gains. Regimes 3 and 4 test whether the mixing problem (specialized text "drowned" by generic text, per Nadir Durrani et al. and MonoCoder) applies to binding as well as to knowledge. Three seeds per cell let us report variance.

We add existing models as an external check, with a caveat about comparability:

  • BioBART and vanilla BART, scored by decoder likelihood.
  • Publicly released continually pre-trained medical decoders and their general-domain parents.
  • PubMedBERT and BERT, scored by pseudo-likelihood. Their scores are reported in a separate column because masked-model scores are not on the same scale as decoder probabilities.

From pair preference to generated hallucination

Pair preference matters only if it predicts errors in generated text. We build item-matched generation probes: each BioCOMPS item has a sibling summarization or case-description prompt that contains the same entities and distractor. We annotate whether the output swaps the attribute onto the distractor. This is done on instruction-tuned open models spanning different amounts of domain exposure, plus our small models after identical light instruction tuning.

The key test is at the level of items within a model, not a correlation across a handful of models. Given a model, does a failure on an item's distracted pair predict an attribute swap on that item's generation sibling better than a failure on its undistracted real-entity pair does?

Baselines

Do nothing. The naive reference is chance (50%) on forced choice. Two heuristic scorers sit alongside it, and any model must clearly beat them:

  • A "pick the nearest entity" scorer. This captures the proximity effect in pure form. A model that matches it is binding by position.
  • A unigram-frequency scorer that prefers the more common entity.

Strongest existing method on this question: the COMPS protocol. We run COMPS itself on all our models, so that BioCOMPS results can be read against the original everyday-domain numbers (about 68% plain, 47.2% with in-between distractors). If BioCOMPS adds nothing beyond COMPS, it is redundant. The evidence that it does add something would be that domain pre-training regimes are ranked differently on BioCOMPS than on COMPS. No reported result exists on Lacuna for any biomedical factual minimal-pair benchmark, so these are new measurements.

Strongest existing task benchmark. BLURB-style and QA evaluations stand in for "domain pre-training helps." For PubMedQA, the file's reference points are 68.1% for fine-tuned BioBERT against a 55.2% majority baseline (PubMedQA). We fine-tune all four of our regimes on PubMedQA with an identical hyperparameter search budget. BioCOMPS beats this baseline if its distractor-drop score predicts generation swaps on held-out items and PubMedQA accuracy does not.

Strongest hallucination-detection competitor: an LLM judge. An LLM judge prompted to flag swapped attributes in the generation siblings is the strongest competitor. The detection numbers to beat are GPT-4 at macro-F1 0.56 on MedHalu and ChatGPT at 58.53% on summarization in HaluEval. The judge sees the same source text and gets the same number of prompt-tuning attempts that we spend tuning our scoring choices.

Ablation isolating the key ingredient. The novel-entity condition is what separates binding from knowledge. We therefore report results with and without it. If real-entity distracted pairs alone predict generation swaps just as well, the knowledge/binding separation is not doing work.

Why not just…?

Why not just use PubMedQA or MedQA accuracy? These benchmarks mix several factors. MedQA shows that retrieval failures are confounded with reasoning failures, with BioBERT-Large at 36.7% on USMLE. In PubMedQA, a majority guess already reaches 55.2%. Neither benchmark manipulates distractor position, so neither can test whether the error comes from binding. We still run them. If they predict item-level generation swaps as well as BioCOMPS does, we will report that the new benchmark is unnecessary.

Why not just compare log-probabilities of true vs. false biomedical sentences without the factorial design? This simple fact-probing approach is the most serious threat, because it costs less and may correlate well with hallucination. Its weakness is specific: a gap in such scores could reflect either missing knowledge or misbinding, which is exactly the distinction in question. The ablation above tests it directly. If undistracted real-entity pairs carry all the predictive signal, the simpler probe wins and we change the claim accordingly.

Why not just measure swaps in generated text with an LLM judge, as MHB does? That measures the outcome but does not explain its cause. Judges also carry their own blind spots: MHB relies on a single judge model, and MedHalu shows that GPT-4 detects at roughly layperson level. We use the judge as a baseline detector and use human annotation as ground truth for the generation siblings.

Why not just fine-tune or prompt a large general model instead of pre-training on domain text? A Comparative Study of PEFT Methods for Python Code Generation finds that base pre-training data predicts downstream competence better than the choice of fine-tuning method. Under our hypothesis, however, neither data nor fine-tuning repairs binding. We include a regime that applies parameter-efficient fine-tuning on PubMed text to the general model. If it closes the distractor drop as well as scratch pre-training does, the practical conclusion is that the cheap route suffices.

Proposed Experiments and Decisive Checks

First check: does the distractor drop change with pre-training data? For each regime we compute the accuracy drop from the no-distractor to the in-between-distractor condition on BioCOMPS. We then fit a mixed-effects model with regime, distractor position, and their interaction, using item and seed as random effects. Our hypothesis predicts:

  • a large main effect of regime on real-entity undistracted accuracy, favoring PubMed regimes;
  • a large main effect of distractor position;
  • a small regime × distractor interaction.

A large interaction in favor of scratch pre-training would refute the hypothesis, and we will report it as such.

Second check: does the drop come from position? We vary the distance between entity and attribute while holding the number of distractors fixed, and we swap distractor order. If the drop comes from position, accuracy should track recency regardless of which entity is the distractor.

Third check: predictive validity. We compare held-out area under the ROC curve for predicting human-annotated attribute swaps in the generation siblings across four predictors:

  • distracted-pair failure;
  • undistracted-pair failure;
  • PubMedQA-tuned confidence;
  • the LLM judge.

We also include a sample of 53 Cochrane reviews from the summarization study, so that the PICO-swap category is tested on naturally occurring text. That study gives no per-model attribute-error rate in the file, so we re-annotate.

Sanity checks. We run our models on BLiMP, where GPT-2 scored 83.0% overall. This confirms that the small PubMed-only models have ordinary grammatical competence, so that BioCOMPS failures cannot be blamed on broken syntax.

If binding failure is confirmed as the dominant cause of swaps and domain pre-training does not repair it, the next step for clinical reliability is training objectives or architectures that target entity–attribute binding, not more domain text.