Teaching Ontology-Grounded Medical QA Models to Survive Missing Entries

Language models answering expert medical questions are often given entries from a curated vocabulary or knowledge graph: definitions and relations such as "drug X treats condition Y" drawn from resources like the Unified Medical Language System (UMLS) or the Chinese Medical Knowledge Graph (CMeKG). This help is real, but it creates a hidden dependency. A model fine-tuned with these entries learns to copy the answer from the lookup whenever the lookup contains it. When an entry is missing, because the drug is new, the patient used a colloquial term, or the direct relation was never curated, the model does worse than one that never had the resource. We propose to fix this during training. While fine-tuning, the model sees ontology context in which the entries it most depends on have been deliberately deleted, and the deletions follow the patterns in which real ontologies actually have gaps. The model then has to learn to reason from neighboring entries, multi-hop relations, and its own knowledge, rather than from an exact match.

The idea goes back to an older result from slot filling. Targeted feature dropout showed that sequence labelers trained with complete city dictionaries memorize the matches and fail on unseen cities, with 80.66% F1 on unseen toloc.city_name in ATIS. Randomly hiding dictionary features for those tokens during training raised this to about 94%. The same failure now shows up at much larger scale in knowledge-graph retrieval. BRINK found that deleting the single direct link a question depends on drops the Hard Hits Rate of these retrieval pipelines on its Family graph from roughly 60–90% to 5–30%, and that retrieval failures outnumber reasoning failures 7:3. On a professional exam, 44% of KMQA's errors on the Chinese pharmacist licensing exam (NLPEC) came from information missing in both text and graph. Our claim is that this collapse comes largely from training, and that a model trained with deletions shaped like real gaps can keep most of its accuracy when the ontology fails it, without losing accuracy when the ontology is complete.

Core Claim and How It Differs From RAFT

The closest prior method is RAFT. RAFT fine-tunes a retrieval-augmented model on passages mixed with distractors and leaves out the golden document in a fixed fraction $P\%$ of training examples, so the model stops over-trusting retrieved text. RAFT works with unstructured passages, removes whole documents uniformly at random, and treats $P$ as a hyperparameter it must tune per domain. Our proposal differs in three specific ways, each of which we test separately.

First, the unit of deletion is a single ontology entry, either a concept definition or a relation, not a document. Second, deletion is targeted. We remove the entries that matter for the answer, meaning those on the path from the question's concepts to the gold answer. Third, the rate of deletion depends on how likely each entry is to be missing in deployment. Rare concepts, recent additions, and terms that can be reached only through colloquial surface forms are deleted more often. Direct relations that can also be reached through a multi-hop rule are deleted often, so that a path around the gap exists. The prediction that would separate this from "RAFT applied to triples" is the following. At a matched overall deletion rate, targeted and gap-shaped deletion should beat uniform random deletion on questions whose entries are naturally missing, not just on questions where we simulated the deletion ourselves.

Research Program

We build a retrieval pipeline over CMeKG for NLPEC and over UMLS for English medical QA. Concept extraction follows the "indirect extraction" step of Yang et al., in which the language model infers latent concepts (for example, "PAD" becomes "Peripheral Artery Disease") that are then mapped to UMLS identifiers. The retriever returns definitions plus relations within two hops of each mapped concept.

For each training question we mark the answer-critical entries. These are entries on graph paths connecting question concepts to answer-option concepts, together with entries whose removal flips the answer of a frozen reference model. We then mine Horn rules with AMIE3, as BRINK does. A Horn rule is a logical pattern of the form "if A relates to B and B relates to C, then A relates to C." These rules tell us which direct relations have indirect substitutes.

Each answer-critical entry $e$ is deleted with probability

$$p(e) = p_0 \cdot g(e),$$

where $p_0$ is a global rate and $g(e)$ is a gap-likelihood weight. The weight $g(e)$ comes from a small classifier trained to separate entries present in a later ontology release from those absent in an earlier one. Its inputs are the concept's corpus frequency, the entry's age, its semantic type, and whether the question reaches the concept only through a non-canonical synonym. The weight is normalized so that the expected number of deleted entries matches the random-deletion baseline.

We need reasoning targets for the training examples that have deletions. A teacher model writes a chain of thought from the reduced context. We keep only chains that reach the gold answer, a rejection-sampling filter in the spirit of Sung et al.'s finding that incorrect training material corrupts domain adaptation. The student, Qwen2.5-7B (the backbone used by RAR²), is fine-tuned on this data mixed with full-context examples.

The analysis tool is a reliance gap. For a test question, we measure accuracy when its critical entries are present minus accuracy when they are deleted. We report this gap broken down by gap-likelihood, so we can see whether the training shrinks it most where real gaps occur.

Evaluation Setting

Every benchmark is split into three coverage groups. The first group contains questions whose critical entries exist in the ontology. The second contains the same kind of questions with those entries deleted at test time. The third, the decisive group, contains questions whose needed concept or relation is naturally absent. We identify this third group by mapping gold rationales or answer concepts to the ontology and finding no entry, and on NLPEC by targeting the missing-knowledge error category that KMQA reported.

The benchmarks are NLPEC with CMeKG, the text subset of MedXpertQA with UMLS, and BRINK's FB15k-237 and Wikidata5m graphs. BRINK serves as a controlled check where deletions are clean and anonymized-entity variants test whether the model actually uses the graph or relies on memorized names. TREC 2017 LiveQA receives physician-style pairwise judgments. It tests whether deletion training also eases the readability loss that Yang et al. observed. The reasoning is that a model less fixated on lookup content may paraphrase less of it, but we treat this as secondary.

Baselines

The naive reference point is the same Qwen2.5-7B with no retrieval and no domain fine-tuning. It tells us how much any ontology use helps. The second baseline is full-context ontology prompting in the style of Yang et al., with the same retriever and no training. That paper reports a 55% versus 20% completeness win rate on LiveQA over base ChatGPT-3.5, which shows the resource helps when its entries exist. The third baseline is supervised fine-tuning with complete ontology context. Our hypothesis predicts this baseline will show the largest reliance gap, mirroring the $p=0$ condition in the slot-filling work.

The strongest competitor is RAFT adapted to ontology context. It uses distractor entries and uniform random omission of all critical entries in $P\%$ of examples, with $P$ tuned on the same validation split and the same compute budget as our method. RAFT reports 35.28% on HotpotQA and 74.00% on HuggingFace APIBench, against 4.41% and 42.59% for domain fine-tuning with retrieval. There is no reported RAFT result found on Lacuna for NLPEC, MedXpertQA, or ontology-structured context, so we will run it ourselves.

We also compare against two published systems. KMQA reports 61.8% NLPEC test accuracy with RoBERTa. RAR² trains a model to write intermediate "thoughts" naming latent clinical concepts and uses those thoughts as search queries over unstructured text. It reports 17.59% tuning-free on MedXpertQA against 14.20% for the plain retrieval system MedRAG, and 60.46% averaged over six medical benchmarks. RAR² represents the option of dropping the ontology entirely. We reproduce it on the same backbone.

The key ablations hold the expected deletion count fixed and vary one ingredient at a time. Random entry-level deletion tests the effect of granularity alone. Targeted deletion with $g(e)=1$ tests targeting without gap-shaping. Gap-shaped deletion of non-critical entries is a placebo. All trained variants use identical data, teacher chains, steps, and tuning trials.

Our method beats the baselines if it outperforms the tuned RAFT-style baseline on the naturally-absent group of NLPEC and MedXpertQA by a margin beyond seed variance (five seeds, paired bootstrap), while staying within one point of full-context fine-tuning on the entries-present group.

Why not just…?

Why not just use a frontier model and skip the ontology? Scale does not close expert gaps on its own. MedXpertQA reports o1 at 49.89% overall despite above 96% on MedQA and MMLU-Med, and TheoremQA reports GPT-4 at 51.0%. Our question is also about a 7B model that can be deployed. We will report a frontier model with no retrieval on the naturally-absent group as a ceiling reference, not as a competitor.

Why not just use RAFT's random golden-context omission? This is the real threat. Random entry-level omission might capture most of the benefit, in which case targeting and gap-shaping add nothing. The matched-rate ablations settle it directly. If random deletion matches targeted deletion on the naturally-absent group, we will report that the useful ingredient is entry-level granularity, and that would be a smaller but still useful finding.

Why not just drop the ontology and retrieve text with generated thoughts, as RAR² does? Text retrieval avoids the dependency on exact entries but brings its own failures. RAR²'s free-form thoughts can name concepts that do not exist, and KMQA's ablations show that plain text alone loses accuracy on licensing-exam questions. We run RAR² on the same backbone in every coverage group. If it wins on the naturally-absent group, ontology grounding is not worth its brittleness in that regime.

Why not just tell the model in the prompt that the ontology may be incomplete? Instructions cannot teach the skill of finding a path around a missing entry. TheoremQA found that handing models theorem statements "did not provide a significant boost," so declarative hints did not translate into procedure there. We include this zero-cost prompt as a baseline on the deleted and naturally-absent groups.

Decisive Checks

The idea depends on three outcomes. The first is whether full-context fine-tuning shows a larger reliance gap than the base model, which would confirm that training creates the dependency. The second is whether gap-shaped targeted deletion shrinks that gap more than matched random deletion on naturally missing entries, not only on simulated deletions. The third, on BRINK's anonymized graphs, is whether the gains come from traversing alternative paths rather than from parametric memory. If the first fails, the problem is not induced by training. If the second fails, RAFT-style uniform omission is sufficient. If the third fails, the model has learned to ignore the ontology rather than to reason around its gaps.