Language models that diagnose rare diseases can reach the right answer for the wrong reasons. They often write a plausible step-by-step explanation, but the explanation does not actually drive the answer. A clinical audit found models at 90% diagnostic accuracy whose reasoning experts rated correct only 10% of the time (Right Prediction, Wrong Reasoning). Training makes this worse. Rewarding only the final answer teaches models to skip or fake intermediate steps, a failure called reasoning collapse. Rare diseases are where this matters most. Most of them resemble common conditions, and a model that learns "these symptoms usually mean the common disease" will be right on average and wrong on exactly the patients who need it. The usual remedy is to check each reasoning step against a trusted verifier, but rare disease diagnosis has no such verifier for free-text patient records.
The idea here is to check reasoning by experiment on the patient record rather than by reading the reasoning text. The model must name the findings it claims are decisive. We then delete those findings from the record and re-ask the model. If the diagnosis does not weaken, the explanation was decorative, and the model loses reward. A second check uses medical ontologies such as the Human Phenotype Ontology and Orphanet, which list the typical findings of each disease and its common lookalikes. From these lists we build "twin" cases in which the findings that separate a rare disease from its common mimic are edited away or swapped. The correct answer for a twin should move to the mimic. This turns one scarce real case into many labeled training and test cases. It also gives a reinforcement learning (RL) signal that rewards reasoning which causally depends on its stated evidence. No symbolic translation of the reasoning chain is needed.
Core Claim
Outcome-only RL, and supervised fine-tuning on curated reasoning chains, both produce rare-disease diagnosers whose stated evidence is often not what drives their answer. Our claim is that a reward built from input interventions fixes this without hurting top-1 accuracy, and it does so beyond what simply adding twin cases as extra training data achieves. There are two such interventions. The first is removing the findings the model cites. The second is removing equally sized uncited findings as a control. Twin cases built from ontology-defined mimic contrasts then supply ground-truth label flips. The distinctive ingredient is the coupling between what the model says mattered and what demonstrably mattered when the record is edited. The twin cases double as a cheap, scalable test for shortcutting, which today requires expert audits of a few hundred records.
Where This Sits
Three findings motivate the proposal. Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning (EHRMIND) shows that RL with rule-checked rewards on health records skips clinical steps unless preceded by a supervised warm-up. Its authors state that the approach does not extend to tasks with thousands of possible diagnoses and no exact-match checker. On the Impact of Fine-Tuning on Chain-of-Thought Reasoning and SCOTT show that fine-tuning and distillation turn rationales into after-the-fact justifications. The same pattern appears in RL for math (Limits of Generalization in RLVR; Slow-Fast Policy Optimization).
Two systems are the closest rare-disease competitors. RareSeek-R1 fine-tunes a 70B model on 17.5k curated reasoning chains with graph retrieval. It never tests whether those chains are faithful, and its authors concede it has no mechanism to rule out common mimics. DeepRare wraps a frontier model in more than 40 tools and reports 95.4% clinician agreement with its reasoning chains, but leaves the model itself untouched.
Two lines of work constrain the design. SymDiag detects unfaithful steps by translating them into Prolog, and suffers translation errors even on math and logic. RareSeek-R1 shows that collapsing narratives into ontology terms drops top-1 accuracy from 0.684 to 0.192, because many decisive signals are labs, imaging and treatment responses. Together these rule out "parse the chain into symbols" and "work only on ontology terms." Our checks therefore act on the narrative text and observe behavior.
Two papers supply methods we reuse. Towards Generalizable Reasoning: GC2PO adds counterfactual rewards to group-relative RL. It perturbs internal representations generically, whereas we perturb clinical evidence selected by medical knowledge. DDxTutor provides an output format in which each clue is marked as supporting or refuting each candidate diagnosis. We adopt that format so the cited evidence is explicit. Mimic pairs come from the RareBench comparative-analysis task (RareBench) and from literature-mined misdiagnosis patterns (Mining Misdiagnosis Patterns from Biomedical Literature). These patterns are exemplified by cases such as Castleman disease resected as sarcoma (Clinical features and outcomes of retroperitoneal unicentric Castleman disease).
Research Program
Output format. The policy reads a patient narrative $x$. It produces a structured differential: a ranked list of diseases, and for the top disease $d$ a set of cited decisive findings $E$, given as spans of $x$ with support or refute labels.
Editing operator. A frozen editor model produces two edited records. In $x_{-E}$ the cited spans are deleted or negated. In $x_{-C}$ an equal number of uncited findings of similar length are removed. Both edits pass through the same editor, so editing artifacts carry no information about which version is the treatment. An extraction check confirms that each edit removed what it claimed and nothing else.
Training reward. Let $p(d \mid x)$ be the policy's normalized confidence in $d$, read from its ranked output scored by a frozen copy of the policy. The reward for a rollout is
$$ r = r_{\text{out}} + \lambda_1 \,\big[p(d\mid x) - p(d\mid x_{-E}) - m\big]_{+} - \lambda_2 \,\big[p(d\mid x) - p(d\mid x_{-C})\big]_{+} - \lambda_3 \,\max(0, |E|-k) $$
The terms are defined as follows:
- $r_{\text{out}}$ is the usual reward for ranking the true diagnosis first.
- The second term rewards citing findings whose removal actually lowers confidence by at least a margin $m$ (necessity).
- The third term penalizes cases where removing uncited findings hurts just as much. This blocks the policy from becoming fragile to any edit.
- The last term caps the citation budget at $k$ spans, so the policy cannot cite the whole record.
The $\lambda$ weights are tuned on a validation split.
Twin cases. For each training case of rare disease $d$ with a known common mimic $d'$, we look up the findings that Orphanet or the Human Phenotype Ontology mark as frequent or obligate for $d$ but not $d'$. We edit those findings out, or swap them for $d'$-typical findings. The twin is labeled $d'$. Only pairs with high-frequency annotations are used. Clinicians spot-check a random sample of twins for medical validity, and we report the rejection rate.
Twins enter training in two ways:
- as outcome-labeled cases;
- through an additional consistency reward that pays when the policy's ranking moves from $d$ to $d'$ between the original and the twin.
Non-phenotypic evidence such as lab trends is often poorly covered by ontologies. For that evidence we rely on the cited-span removal test alone and say so in the results.
Training pipeline. We follow the EHRMIND recipe: a small supervised warm-up on DDx-format reasoning chains, then group-relative policy optimization with the reward above. Models are 3B and 8B open reasoning models, which keeps several seeds affordable. Each rollout needs two extra scoring passes, so we expect overhead on the order of GC2PO's reported 1.2x over plain GRPO, and well under 2x.
Baselines
Naive reference. The same open model, prompted zero-shot in the DDx output format. This calibrates how much any training helps and how unfaithful an untrained model already is.
Strongest method at matched scale. The EHRMIND recipe, meaning supervised warm-up followed by outcome-only GRPO, run on the identical training cases, model, rollout count, and tuning budget. It is the strongest competitor because its authors show it already prevents the step-skipping form of collapse in clinical RL. If it also yields faithful citations, our reward is unnecessary. We found no reported result on Lacuna for EHRMIND on rare disease diagnosis. Its only reported clinical number, 51.96% on MedCalc-Bench, is a different task. All of its rare-disease numbers will therefore come from our own runs.
Curated-chain fine-tuning. Supervised fine-tuning on curated reasoning chains in the style of RareSeek-R1, at matched scale and data.
Generic counterfactual RL. GC2PO, with its counterfactual reward swapped in for ours under the same compute.
Reference points with published numbers. These are for context, not matched comparisons:
- GPT-4 reaches 52.0% top-1 on the 75-case RareBench head-to-head subset, against 44.7% for senior specialists (RareBench).
- DeepRare reports 57.18% Recall@1 on its phenotype-based tasks (DeepRare).
- RareSeek-R1 reports 0.684 top-1 on internal narratives, a set that is not public.
Ablations isolating the key ingredient. Two ablations matter most:
- Augmentation-only: twins added purely as extra outcome-labeled data, with no intervention reward.
- No-control: our reward with the uncited-removal penalty dropped.
What counts as winning. Beating the strongest baseline means meeting all three conditions:
- top-1 accuracy within one point of the EHRMIND recipe, or better;
- a significant gain on the necessity and twin-flip metrics, with confidence intervals over three seeds;
- a clinician-rated reasoning correctness rate higher on a blinded sample.
Why not just…?
Why not just use the EHRMIND recipe, warm-up plus outcome RL? Its evidence for avoiding collapse is measured by whether steps are skipped, not by whether the written steps drive the answer. The audit in Right Prediction, Wrong Reasoning shows that long, complete-looking chains can still be decorative. This alternative is a direct baseline. If it matches our necessity and twin-flip scores, we will report that warm-up suffices.
Why not just add the twin cases as training data? This is the most serious threat. Augmentation alone may teach the model to separate rare diseases from their mimics and recover most of the accuracy gain. The augmentation-only ablation settles it. Our prediction is that augmentation improves twin-flip accuracy but not the necessity score on real, unedited cases. That is because a model can learn to read the twin-defining findings while still citing other things. If augmentation matches us on necessity as well, the contribution reduces to a data-generation method plus a test set, and we will present it that way.
Why not just use a judge model or reviewer agents to score reasoning steps? In the rheumatoid arthritis audit, adding a reviewer agent raised accuracy without fixing flawed reasoning. SymDiag reports that LLM-as-judge detects reasoning defects with F1 62.8, barely above answer matching at 57.7. A judge-scored process reward would therefore likely be gamed. We include a judge-reward variant at matched compute to measure this directly.
Why not just use a frontier model or an agent system like DeepRare? These are strong at inference time. But RareSeek-R1 reports OpenAI o1 at 0.487 against 0.684 for a domain-trained model, and agent systems leave the model's own shortcut behavior unchanged. Our twin test can be run on these systems through their APIs, so we will report their shortcutting rates as an external measurement.
Proposed Experiments
Data. Training uses Phenopacket-Store (5,213 records) and the public RareBench cohorts (MME, LIRICAL, HMS, RAMEDIS). Evaluation uses a held-out split in which 20% of diseases never appear in training, which probes ultra-rare generalization.
Metrics. We report six:
- top-1 and top-10 recall;
- the necessity score, $p(d\mid x)-p(d\mid x_{-E})$;
- the control gap, $p(d\mid x)-p(d\mid x_{-C})$;
- twin-flip accuracy on held-out twins;
- accuracy on RareBench's comparative-analysis pairs;
- the share of blinded chains rated Correct, Minor Flaw or Major Flaw by two clinicians, using the categories from the rheumatoid arthritis audit, on 200 sampled cases per system.
Decisive checks. There are five:
- Does augmentation-only match our method on necessity on unedited cases?
- Does the intervention reward cost accuracy, especially on diseases absent from training?
- Is the reward being gamed? Signs would be citation sets growing to the cap, or the control gap rising.
- Does the necessity score computed automatically agree with clinician ratings? If it does not, the cheap test is not a valid proxy and the training signal is suspect.
- Do models trained without our reward show larger accuracy drops on twins than on matched originals? That would confirm prevalence shortcutting toward the common mimic.
References
- Right Prediction, Wrong Reasoning: Uncovering LLM Misalignment in RA Disease Diagnosis
- Training LLMs for EHR-Based Reasoning Tasks via Reinforcement Learning
- On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
- SCOTT: Self-Consistent Chain-of-Thought Distillation
- Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
- Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
- A Specialized Large Language Model for Clinical Reasoning and Diagnosis in Rare Diseases
- An agentic system for rare disease diagnosis with traceable reasoning
- SymDiag: Explainable Diagnosis for LLM Reasoning via Neuro-Symbolic Verification
- Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning
- DDxTutor: Clinical Reasoning Tutoring System with Differential Diagnosis-Based Structured Reasoning
- RareBench: Can LLMs Serve as Rare Diseases Specialists?
- Mining Misdiagnosis Patterns from Biomedical Literature.
- Clinical features and outcomes of retroperitoneal unicentric Castleman disease resected as sarcomas: insights from a high-volume sarcoma center