Near-Miss Distractors: Does Biomedical Pre-Training Make Models Easier to Fool?

When a medical question-answering system retrieves evidence, the most dangerous wrong passage is usually not an unrelated one. It is a passage about the right kind of thing but the wrong instance: a trial of a sibling drug in the same class, a finding about the parent disease category rather than the specific subtype, a gene from the same family. We call these ontologically near distractors, meaning passages whose central concept sits one or two steps away from the question's concept in a curated biomedical hierarchy such as MeSH or SNOMED-CT. We propose that domain-specific pre-training, which means continuing to train a general language model on medical literature, may make models more susceptible to these near misses. The reason is that such pre-training teaches the model that related concepts are similar, and a model that has learned "metoprolol and atenolol are nearly the same thing" may treat a passage about one as evidence about the other. No one has measured this. The research file identifies the question as open: nothing tests whether medical pre-training strengthens or weakens resistance to retrieval noise.

The proposal has two parts. The first is a controlled measurement. We generate distractors at graded distances in a biomedical ontology and compare a general model against its medically pre-trained descendant built on the same base. The second is a targeted repair. We run a short continued-training stage that uses the ontology to build hard contexts: the correct passage is shown next to sibling-concept passages, and the model is trained to answer from the exact concept only. The key test is whether distractors chosen by ontological distance teach something that ordinary retriever-mined hard negatives do not.

Core Claim

The claim is that noise sensitivity in biomedical models is structured by the ontology. A model's error rate on a distractor should depend on how far that distractor's concept sits from the question's concept in the hierarchy, and not only on how similar the text looks. Medical pre-training should steepen this dependence. We expect it to reduce errors from unrelated distractors, because the model understands the question better, and to raise errors from sibling and parent-level distractors, because the model conflates nearby concepts. If this holds, the fix belongs in training data built from the ontology, not in generic noise-robustness training.

This matters because three lines of prior work each touch the problem without covering it. BioLORD (BioLORD) uses contrastive learning over UMLS definitions to produce embeddings that deliberately place parents and children close together, and it beats SapBERT on SnomedCT leaf-to-parent mapping (37.0% vs 29.1%). That closeness is exactly the property we suspect becomes a liability inside a generator reading noisy context. BioLORD was only ever tested on clean concept pairs, never in a generation setting. Llama-3-Meditron (Llama-3-Meditron) and BioBART (BioBART) show that continued medical pre-training improves clean benchmark accuracy. They do not report robustness to retrieved distractors. Meditron matters especially here because it shares a base with Llama-3.1-8B, which gives an unusually clean comparison for isolating the effect of pre-training. Ext2Gen (Aligning Extraction and Generation for Robust RAG) trains a model with preference optimization to extract supporting sentences before answering, using up to 25 distractor chunks drawn from a vector database, and it includes PubMed data. Its distractors are chosen by embedding similarity, with no notion of ontological distance, and it uses a general instruction-tuned model. Our method reuses Ext2Gen's training recipe and changes what counts as a hard negative.

Research Program

The first component is a benchmark we call OntoDistract, built on PubMedQA (PubMedQA) and on MedQA with textbook retrieval (What Disease does this Patient Have?). For each question we link the pivotal concept to a MeSH or SNOMED-CT identifier. We then collect PubMed abstracts whose pivotal concept falls into one of four bands:

  • the same concept;
  • a sibling (sharing a parent);
  • the parent or an ancestor;
  • an unrelated concept from the same broad category.

Within each band we also control the direction of the distractor's conclusion. It either agrees with the gold answer or is misleading, meaning it supports the opposite yes/no answer. Misleading distractors let us test the "rationalization trap" in medicine. That failure was documented on general science by Thinking in a Crowd: Qwen3-8B with step-by-step thinking fell from 96.22% to 77.03% on SciAux when given two misleading snippets.

Linking concepts to identifiers is a known weak point. The best system on BioCreative V Track 4 reached only 20% F-measure on full-statement extraction without gold entities (BioCreative V track 4). We therefore do not trust automatic linking alone. We combine dictionary matching with BioLORD nearest-neighbour search, and a clinician-annotated subsample of several hundred items will verify both the band labels and whether each distractor is truly non-answering.

The second component is the measurement. For a model $m$ and distractor band $b$, we define the robustness gap $\Delta_m(b) = \mathrm{Acc}_m(\text{clean}) - \mathrm{Acc}_m(b)$: the accuracy lost when a band-$b$ distractor is added next to the gold passage. We also record the adoption rate, which is the fraction of errors whose answer matches what the misleading distractor implies. The central quantity is the difference in gap between the medically pre-trained model and its general base, $\Delta_{\text{Meditron}}(b) - \Delta_{\text{Llama}}(b)$, plotted against band.

We control for text similarity by matching distractors across bands on their embedding similarity to the question. This lets us attribute any trend to ontological distance rather than to lexical overlap. We run this for both the 8B pair and the BioBART/BART pair, and we run it with thinking-style prompting both on and off.

The third component is the repair, which we call ontology-contrastive continued training. We build training contexts in which the gold passage appears among sibling and ancestor passages, balanced on text similarity, together with extraction targets as in Ext2Gen. We then run the same preference-optimization stage as Ext2Gen on both the general base and the medically pre-trained base. The comparison is against an identical stage whose negatives are retriever-mined hard negatives matched on embedding similarity and token budget. Only the principle for choosing negatives differs.

Baselines

The naive reference point is each model reading the gold passage plus distractors with a standard prompt and no robustness training. The same model with the gold passage alone gives the clean ceiling. For PubMedQA the file reports a majority-class accuracy of 55.2%, fine-tuned BioBERT at 68.1%, and human experts at 78.0% (PubMedQA). These numbers anchor what a clean score means. We have no reported result found on Lacuna for Llama-3-Meditron-8B's exact PubMedQA accuracy; the file states only that it exceeds other Llama-3.1-8B variants by over 3%. We will measure it ourselves.

The strongest existing method is Ext2Gen-R2, which uses preference optimization to make the model extract supporting sentences and then answer. It is the strongest competitor because it already trains the generator itself, rather than an external filter, against realistic retrieved distractors, including PubMed ones. It also outperformed pre-generation compressors such as Recomp, CompAct and EXIT. On Llama-3.1-8B-Instruct under noisy retrieval it raised the average QA score from 0.473 to 0.557 (Aligning Extraction and Generation).

We will re-run Ext2Gen-R2 on our base models using its own recipe. It gets the same number of training contexts, the same preference pairs per question, the same number of GPU hours, and the same hyperparameter search budget as our method. Beating it means lowering the sibling-band and ancestor-band robustness gaps by a margin outside the bootstrap confidence interval, without raising the unrelated-band gap and without reducing clean accuracy by more than one point.

The ablation that isolates our key ingredient is the matched hard-negative run described above. It is Ext2Gen training in which the negatives are retriever-mined passages chosen to match the embedding-similarity distribution of our ontology-chosen ones. If this ablation closes the gap as well as our method does, then ontological distance adds nothing beyond "hard negatives," and we will report exactly that.

A post-hoc checker is the third comparison point. MedRAGChecker (MedRAGChecker) splits an answer into atomic claims and checks each against text entailment and the Drug Repurposing Knowledge Graph. It reports 87.4% accuracy on three-way claim verification with a distilled 8B student. We will apply it to the naive model's outputs and measure how many sibling-band errors it flags.

Why not just…?

Why not just use Ext2Gen with retriever hard negatives, which are already semantically close? This is the real threat. Dense retrievers do surface sibling-concept passages, so ordinary hard-negative mining may already cover the near-miss case. Our reason for doubting it is that embedding similarity mixes together passages that merely share vocabulary and passages about a confusable concept. Only the second kind targets the conflation we hypothesize. The matched hard-negative ablation settles the question directly. If it performs as well as ontology-chosen negatives, the contribution becomes the measurement study plus a recommendation to use mined negatives, and we will say so.

Why not just filter passages whose linked concept differs from the question's before generation? For sibling distractors an identifier-matching filter may work well, and we include it as a baseline. Its weakness shows up at the ancestor band, where a parent-level claim ("beta-blockers reduce…") is sometimes valid evidence and sometimes not. A filter that drops such passages discards real evidence, while one that keeps them lets the distractor through. Its accuracy also depends on concept linking, which the BioCreative results show is brittle. We will report the filter's accuracy per band, including on gold passages that it wrongly removes.

Why not just verify answers afterwards with a knowledge-graph checker? MedRAGChecker can flag a wrong claim, but it cannot produce the right answer. Its authors also report that when the graph lacks an entity, as with rare conditions or new drugs, it falls back to text entailment. On decisions where the two signals disagreed, text entailment alone matched clinicians only 63.4% of the time, against 69.8% for the fused checker. Sibling drugs in the same class are exactly where graph edges are sparse and text entailment is easily fooled. The flag-rate comparison will show how much of the near-miss error it catches.

Why not just instruct the model to ignore passages about other concepts? Instructions alone did not stop misleading-context failures on SciAux (Thinking in a Crowd). Distraction-aware answer selection (Improving Zero-shot Reader) needed a query-likelihood score on top of its instructions. We still include a concept-explicit instruction ("answer only about X; ignore passages about related but different X′") as a cheap baseline in the measurement grid.

Decisive Checks

The measurement result would be decisive if the medically pre-trained model's gap relative to its base is negative on the unrelated band and positive on the sibling and ancestor bands, with text similarity held fixed, on both the 8B pair and the BART pair. That pattern would confirm that pre-training trades general distraction resistance for near-miss vulnerability. A flat difference across bands would refute the core claim, and we would report pre-training as neutral with respect to noise.

The repair result would be decisive if ontology-contrastive training reduces the sibling-band adoption rate more than both Ext2Gen-R2 and the matched hard-negative ablation, with equal budgets. It should also help the medically pre-trained model more than the general base, since the structured conflation the repair targets should be stronger there.

Finally, we will test transfer. Models trained only on PubMedQA-derived contexts will be evaluated on MedQA textbook retrieval and on BioASQ snippets with sibling distractors injected. The purpose is to check that the gain reflects a learned habit of distinguishing concepts, not memorized question formats.