Counterfactual Case Twins for Separating Recalled Diagnoses from Clinical Reasoning

Frontier language models now reach correct diagnoses on famously hard published medical cases. When their written reasoning is turned into structured graphs, though, that reasoning looks almost random: clinically similar cases produce reasoning no more alike than unrelated ones. There are two readings of this. One is that models reason clinically but without stable habits. The other is that they recall the published answer and then write a plausible justification around it. These readings call for opposite responses, and current evaluations cannot tell them apart. The cases are public, so memorization cannot be ruled out. The scoring is also mostly about the final answer, so recall and reasoning earn the same credit.

We propose a direct test. For every published case we build counterfactual twins: physician-edited versions in which one or two decisive findings are changed so that the correct diagnosis moves to a different disease on the original differential, which is the list of candidate diagnoses the case discussion considered. We also build neutral twins, which change surface details but keep the diagnosis. A model that reasons should follow the edited evidence to the new answer. A model that recalls should keep giving the published diagnosis, and its reasoning graph should fail to connect the changed finding to its conclusion. We then run the same construction on cases published after the models' training cutoffs, where recall is impossible. Comparing the two sets separates memorization from a more ordinary failure to use evidence. Scoring each response on several dimensions (outcome, evidence use, calibration, and cross-case consistency) shows which dimensions contamination inflates.

Core Claim

The claim is that contamination inflates outcome scores and leaves process scores alone, and that counterfactual twins measure this case by case. Clinical Reasoning Graphs converts free-text diagnostic reasoning into graphs whose nodes are clinical features and diagnoses and whose edges are relations such as "supports" and "argues against." Across 50 NEJM clinicopathological conference cases, within-cluster graph similarity was 0.475 and between-cluster similarity was 0.472, and the authors could not say whether memorization was responsible. This study supplies the experiment that settles that question. If recall drives the null result, three things should happen. On pre-cutoff counterfactual twins, models should stick to the published diagnosis far more often than they do on post-cutoff twins. Their graphs should give the edited finding little or no edge into the final diagnosis. And outcome accuracy on original cases should exceed accuracy on matched twins by more than any loss in evidence-use scores. If none of this appears, the "competence without consistency" result describes how these models reason clinically, and is not an artifact of leaked cases.

Research Program

The central estimator is a difference-in-differences anchoring score. For a model $m$, let $A_{\text{pre}}(m)$ be the fraction of counterfactual twins of pre-cutoff cases on which $m$ outputs the original published diagnosis rather than the new correct one. Let $A_{\text{post}}(m)$ be the same fraction on twins of post-cutoff cases. Then

$$\Delta_{\text{recall}}(m) = A_{\text{pre}}(m) - A_{\text{post}}(m).$$

$A_{\text{post}}$ captures generic insensitivity to edited evidence: stubbornness, ignoring a lab value, or edits that are too subtle. Recall cannot exist on post-cutoff cases, so whatever anchoring remains above that floor is attributable to recall. The same differencing is applied to graph-level quantities, in particular the fraction of twins in which the edited finding appears as a node with a "supports" or "argues against" edge into the final diagnosis.

Building the twins is most of the work. We start from the 50 NEJM cases and 20 disease clusters used by Clinical Reasoning Graphs. For each case, two physicians independently pick an alternative diagnosis that the original discussion considered and rejected. They then make the smallest set of edits (a lab value, an exposure history, an imaging result) that makes the alternative correct. A third physician blind-checks every twin for clinical coherence, for a single defensible answer, and for difficulty on a five-point scale relative to the original. Twins that differ in difficulty from the original by more than one point are dropped. This difficulty matching answers the main objection to temporal splits raised by Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?, namely that new test items can differ from old ones in more than exposure. Neutral twins change names, demographics within clinically irrelevant ranges, sentence order, and wording. This follows the meaning-preserving rewrites of Meta Probing Agents but is done by humans, avoiding that paper's reliance on a frontier model as rewriter.

We target about four counterfactual and two neutral twins per case. The same protocol is applied to as many NEJM cases as exist that were published after the documented cutoffs of the open-weight models we test. Because that pool is small and shrinks with every model release, it serves as the calibration arm, not the main test set.

Every response is scored along four dimensions. The goal is to replace a single correctness number with a profile, in the spirit of the multi-dimensional rubrics in education research such as Exploration on Teaching Reform of SPSS Data Analysis. Outcome is whether the final diagnosis is correct. Evidence use is whether the graph routes the decisive findings correctly. Calibration is whether stated confidence tracks correctness. Consistency is within-cluster versus between-cluster graph similarity. Graph extraction reuses the CRG ontology and pipeline. Two physicians also grade a stratified 20% sample on a point checklist and on an overall reasoning judgment, which borrows the dual-rubric design of Evaluating AI Grading on Real-World Handwritten College Mathematics. This lets us check that automated evidence-use scores agree with clinicians.

Models fall into two tiers. The frontier API models from the CRG study are evaluated on behavior only. Open-weight models, such as the Qwen2.5 family used by DCR and GPT-OSS-120B as used in BeyondBench, also give access to token probabilities and hidden states, so membership and mechanistic signals can be compared against $\Delta_{\text{recall}}$ case by case.

Baselines

The naive reference point is the CRG analysis as published: accuracy and graph consistency on the original 50 cases with no contamination control. It defines the effect to be explained, a within-cluster similarity of 0.475 against a between-cluster similarity of 0.472, reported by Clinical Reasoning Graphs. We rerun it on our models with identical prompts.

The strongest perturbation competitor is a meaning-preserving rewrite of each case, as in Meta Probing Agents. That method reports a 15.7% accuracy drop for GPT-4-Turbo on fully probed MMLU, but no reported result was found on Lacuna for clinical vignettes. It is the strongest competitor because it is cheap and already deflates contaminated scores. We run it on the same cases and models, using our neutral twins as its human-verified implementation so that rewrite quality is held fixed. Our method beats it if counterfactual anchoring flags recalled cases that neutral-twin accuracy drops miss, judged against the post-cutoff calibration arm.

The strongest detection competitor is a membership signal computed from the model's own probabilities: a Min-K% score over each case's text, the filter used by C2LEVA. Alongside it, we compute the 37-feature recall-versus-reasoning classifier of RADAR, which reports 93.0% accuracy, but on only 100 test examples. No reported result was found on Lacuna for Min-K% on clinical cases. Both are run on the open-weight tier with identical inputs. Our method beats them if $\Delta_{\text{recall}}$ predicts physician-judged unsupported diagnoses better than either score, or if the two scores disagree with anchoring in ways the post-cutoff arm shows anchoring gets right.

The key ablation removes the post-cutoff difference and reports raw $A_{\text{pre}}$ alone. If raw anchoring explains the same things as $\Delta_{\text{recall}}$, the calibration arm adds nothing and the method reduces to a counterfactual robustness test.

Why not just…?

Why not just evaluate on CPC cases published after the training cutoff? This is the most serious threat, and it is built into the design. MLAgentBench shows how revealing temporal splits are: agents reach 100% on a classic Kaggle task and fall to near zero on post-cutoff challenges. A temporal split has two weaknesses for this question, though. It cannot attribute recall to individual cases, and it mixes exposure with changes in case difficulty and topic. If CRG consistency on post-cutoff cases simply matches the original null result and pre-cutoff anchoring adds no information, then the temporal split answers the CRG question by itself and we will say so. What would remain is a per-case contamination audit that keeps working after the post-cutoff pool is absorbed into later training data.

Why not just run Min-K% or another membership test on the case text? A membership score shows that a model has probably seen a text. It does not show that the model's answer depends on having seen it. NEJM cases may be memorized and still be reasoned through. The comparison in the Baselines section tests this directly. If Min-K% scores alone predict anchoring well, the cheaper tool should be preferred for screening, and twins kept for confirmation.

Why not just paraphrase the cases and measure the accuracy drop? A paraphrase keeps the answer, so a model that remembers the case's meaning can still retrieve the diagnosis. Meta Probing Agents shows that rewrites lower scores but cannot show what the lowered score is made of. Only a change that moves the correct answer can show a model holding on to the old one. Neutral twins are included so that this claim is tested, not assumed.

Why not just apply a post-hoc deflation factor such as DCR? DCR adjusts accuracy to within 4% of uncontaminated baselines, but it was validated on SST-2, LIAR2, and GSM8K using fixed fuzzy rules. It produces a single discounted number and says nothing about which reasoning dimensions are inflated, which is the question here.

Proposed Experiments and Decisive Checks

The first check is whether $\Delta_{\text{recall}}$ is reliably positive for each model, using bootstrap confidence intervals over cases. The second is whether evidence-use scores on counterfactual twins fall below those on neutral twins for pre-cutoff cases but not for post-cutoff ones. The third is whether within-cluster minus between-cluster graph similarity, measured on neutral twins of post-cutoff cases, stays near the published null. If the gap stays near zero there, inconsistency is a property of how these models reason. If it widens, the CRG result was largely a contamination artifact. The fourth check asks whether the outcome-score inflation on original cases, relative to matched twins, exceeds the inflation in evidence-use and calibration scores. That would confirm that contamination distorts the outcome dimension of a multi-dimensional profile more than its process dimensions. The fifth is whether case-level $\Delta_{\text{recall}}$ correlates with Min-K% and RADAR scores on the open-weight tier, which tells us whether cheap detectors could stand in for twin construction in future audits.

Each outcome is informative. A positive $\Delta_{\text{recall}}$ together with recovered consistency means public clinical case collections overstate diagnostic reasoning. A null $\Delta_{\text{recall}}$ means the "competence without consistency" result is about the models and should be addressed as a reasoning problem.