Twin Clinical Paths: Scoring Long-Form Medical Answers Against the Graph That Generated the Question

We do not have a trustworthy way to test whether a language model can reason through specialist medicine in its own words. Static exam sets saturate and leak into training data: one model family exceeds 96% on undergraduate-level MedQA yet reaches only 49.89% on specialty board questions. Free-text answers, meanwhile, are scored either by word-overlap metrics that ignore facts or by language-model judges that favour text resembling their own. This proposal fixes both problems with one construction.

Each test question is generated from a sampled multi-step path through a medical knowledge graph, a database of facts stored as (subject, relation, object) triples such as (drug, contraindicated_for, disease). Because we built the question from a known path, the path is also the answer key. A free-text answer is scored by extracting the facts it asserts and checking whether they retrace that path, with no reference prose and no judge's opinion involved. Each question also gets a "twin": an otherwise identical version in which one link of the path is replaced by a fictional entity whose properties are stated in the prompt. Twins have no memorized answer, so the score gap between a question and its twin measures how much a model relies on recall rather than reasoning.

Core Claim

When a benchmark item is generated from a graph path, three things that are usually separate become one object: the question, the gold reasoning trace, and the scoring rule. That unity buys three things at once. First, an evaluation of long-form clinical answers that needs no reference text and no model judge. Second, label correctness by construction, a known weakness of rewriting-based anti-contamination methods. Third, a per-item measurement of memorization through matched twins.

The claim is testable. Path-recovery scores on twin pairs should agree with physician judgments of answer correctness and reasoning at least as well as the strongest judge-based evaluator. They should also expose a "right answer, wrong path" failure mode that final-answer accuracy hides.

How this differs from the closest work

Four lines of prior work each hold one piece.

Two papers motivate the scoring design but stop short of it. MedRAGChecker splits biomedical answers into atomic claims and checks each one with textual entailment plus graph embeddings over the Drug Repurposing Knowledge Graph. Knowledge-Graph Based RAG System Evaluation Framework shows that atomic checks miss broken links between individually true facts, and it scores multi-step reachability with shortest-path search. Neither has a known gold path. MedRAGChecker verifies whatever the answer happens to say, and Dong et al. infer connectivity on open-domain Wikipedia text. We get the gold path for free from the generator and work over safety-critical relations such as contraindications.

A second group supplies the generation idea but works in a different regime. DyVal generates reasoning problems from synthetic directed acyclic graphs and exposes memorization when description order is perturbed. KBE-DME evolves visual questions by expanding knowledge subgraphs. CoreEval swaps in post-cutoff news triples for classification tasks. All three score discrete answers by exact match, and none uses a specialist ontology. We generate open-ended clinical questions and score free text.

A third paper supplies the twin mechanism but in a form we must relax. LARK replaces every entity name with an anonymous identifier so the model must reason over graph structure rather than memory. That also strips out the domain meaning a clinician uses. Our twins anonymize exactly one node and keep the rest of the clinical language, so the reasoning task stays medical while one link becomes impossible to recall.

Finally, the generator's substrate is PrimeKG. It has 129,375 nodes and 4,050,249 edges, covers 17,080 diseases, and includes over 61,000 contraindication edges. That coverage is what makes multi-step pharmacology paths with built-in safety constraints possible.

Research Program

Item generation. We sample short paths, two to four links long, from PrimeKG under clinically meaningful templates. One example runs from a patient condition, to a disease, to candidate drugs, to a filter through contraindication edges that excludes some of them. Another runs from a gene, to a phenotype, to a disease, to an indicated therapy. A model then turns each path into a short clinical vignette and question. Because the answer is computed from the graph, and not written by the model that phrases the vignette, the label cannot drift during rewriting. That is the failure mode CoreEval flags for synthetic rewriting. Clinicians rate a sample of vignettes for naturalness and for agreement with the computed answer, so we measure whether PrimeKG errors propagate into labels.

Twin construction. For each item we choose one intermediate node, usually a drug, and replace it with a fictional compound. The vignette states the compound's relevant properties in plain clinical language, for example "Compound K-17 is a strong inhibitor of the enzyme that clears drug X." The graph walk is then recomputed so that the twin's correct answer follows from the stated facts. Real and twin items share wording, depth, and template, so a model that reasons should score about the same on both. A model that recalls associations should drop on the twin, or answer the twin with the memorized real-world answer.

Scoring. An extraction model pulls asserted triples from the free-text answer and links them to PrimeKG identifiers, or to the fictional entity, using names and stated properties. Let $P$ be the set of gold path edges, $A$ the set of linked edges the answer asserts, and $C \subseteq A$ the asserted edges that PrimeKG or the twin's stated facts contradict, such as recommending a drug across a contraindication edge. The path score is

$$S = \frac{|A \cap P|}{|P|} - \lambda \frac{|C|}{\max(|A|,1)}$$

Here $\lambda$ weights safety contradictions; it is fixed before any evaluation and reported in a sensitivity sweep. Asserted edges that are neither in PrimeKG nor contradicted by it are reported separately as "unverifiable" rather than penalized. This keeps the graph's incompleteness from counting against correct answers, the coverage problem MedRAGChecker documents.

Memorization gap. We define $G = \bar S_{\text{real}} - \bar S_{\text{twin}}$ as the gap between mean path scores on real and twin items, reported per model and per template. As a secondary analysis, we compare base and chat-tuned versions of the same model family on $G$. Beyond Benchmarks reports an "alignment tax": chat tuning reduces specialist recall. If that holds here, chat models should lose on real items but not on twins. If chat tuning also damages the ability to follow stated facts, the twin score would fall too.

Extractor bias. The extractor is a model, so it could reintroduce the self-preference documented in LLMs as Narcissistic Evaluators. We give it a deliberately narrow job, linking stated facts to a closed identifier set rather than judging quality. We use extractors from at least two families that are not among the evaluated models, and we report agreement between them on every answer.

Baselines

The naive reference is final-answer accuracy on the same generated items: does the answer name the graph-computed drug or diagnosis? It costs almost nothing, and if it ranks models the same way as $S$ and agrees with physicians as well, the path scoring adds nothing.

The strongest metric competitor is a reference-guided judge. A frontier model receives the question, the gold path written out in prose, and the candidate answer, and grades correctness and reasoning on a rubric. This is the most dangerous alternative because it has the same privileged information our metric has. The narcissism paper also finds that self-bias is much milder when a reference is supplied. We run it with the identical gold path and the same items. We tune its prompt on the same development split and with the same budget that sets $\lambda$ and the extractor prompt, and we use a judge family distinct from the evaluated models. No reported result was found on Lacuna for reference-guided judges on medical long-form answers.

The strongest existing domain verifier is MedRAGChecker run unchanged on our answers. Its paper reports that its fused output agreed with human experts 69.8% of the time on flipped decisions, against 63.4% for entailment-only checking, and that its student ensemble reached 87.4% accuracy on three-way entailment. The Dong et al. path metric, rebuilt over PrimeKG without the gold path, isolates whether knowing the gold path matters. It has no reported numeric agreement on Lacuna, only near-zero scores on deliberately wrong answers.

The ablation that isolates our key ingredient is $S$ computed with the gold path removed. That variant keeps only contradiction checks and reachability, as in the prior verifiers. We beat the field if $S$ shows higher rank correlation with physician scores than both the reference-guided judge and MedRAGChecker, with non-overlapping bootstrap intervals. On cost and determinism alone we would claim only parity.

Why not just…?

Why not just let a frontier model judge the answer, given the gold path? This is the real threat, and we treat it as a hypothesis to test rather than a strawman. Its known weaknesses are self-preference and verbosity bias, both documented in LLMs as Narcissistic Evaluators, and sensitivity to prompt wording, reported in Rethinking LLM-as-a-Judge. The physician study settles it. We stratify agreement by evaluated-model family and by answer length. If the judge matches $S$ overall and shows no family or length effect, we will report that a judge given the gold path is sufficient. The contribution then reduces to the twin-pair benchmark plus a cheaper, auditable scorer, and we will say so.

Why not just ask multi-step graph questions in multiple-choice form? Multiple choice is cheaper and exact. However, it rewards correct guesses reached through memorized associations, the pattern behind the 33.8% of GPT-4 failures in DyVal labeled "unsubstantiated responses." It also cannot detect an unsafe intermediate claim. Comparing $S$ with final-answer accuracy measures how often the right answer arrives by a wrong or contraindicated path. If that rate is negligible, the long-form scoring is unnecessary.

Why not just paraphrase existing exam questions or use a newer static set? Paraphrasing keeps the memorized relational core, as Rethinking Benchmark and Contamination for Language Models with Rephrased Samples shows. A new static set such as MedXpertQA is hard today but will be scraped like MedQA was. Twins differ from both because the answer to a twin cannot appear anywhere in the training data. We include paraphrased MedQA items in the study. If their score drop matches the twin gap, twins add little.

Proposed Experiments and Decisive Checks

We will generate about 2,000 real–twin pairs across four templates. The generated set also includes, for calibration, a slice of MultiMedQA long-form questions and paraphrased MedQA items. Evaluated models span open base and chat pairs from at least two families, a medically tuned model, and frontier systems. For context on saturation, the file reports 86.5% on MedQA for Med-PaLM 2 and above 96% for o1-preview.

Three physicians will independently grade 600 answers, balanced across real and twin items and across model families. They score final correctness, reasoning validity, and safety, following the axis-based rating style of Med-PaLM. Metric validity is rank correlation with the physician mean, broken down by family and length.

The decisive checks are three. First, on real items, does $S$ match or exceed the reference-guided judge in agreement with physicians, and does it beat MedRAGChecker and the no-gold-path ablation? Second, is the twin gap $G$ clearly nonzero for models known to be strong on static exams? It should be larger than the drop on paraphrased MedQA, and comparable in direction to the 5.42% average drop for proprietary models reported by CoreEval. Third, does $G$ correlate with token-level recall on the original items? That last check is the near-certain prediction signal used by Yax et al., and it would confirm that twins measure memorization rather than mere unfamiliarity with fictional names. A control in which the fictional compound copies a real drug's properties exactly separates name unfamiliarity from loss of recall.

If $G$ is near zero everywhere, or if physicians judge twins as clinically incoherent, the memorization claim fails.