Does the Model That Writes the Recommendation Agree With the Model That Checks It? Scope Consistency for Single-Case Clinical Evidence

A clinical case report describes what happened to one patient. It shows that an outcome is possible. It cannot show that the outcome is typical, or that a treatment works across a population. Language models are increasingly asked to summarize and act on such reports. We propose to test whether a model keeps that limit straight in two different modes. In the first mode the model writes a recommendation from the report. In the second it judges whether a given claim is warranted by the report. The open question is whether these two modes agree on how far one patient's outcome can be stretched. Earlier work on the "generator–validator gap", the finding that a model's free-text answers and its own true/false judgments often disagree, suggests they may not. That gap has only been measured on arithmetic, word relations and trivia, never on reasoning about how strong a piece of evidence is.

The proposal has three parts. We build a benchmark of claims at graded scope for real case reports. We measure the gap between what scope a model implies when it writes and what scope it accepts when it judges. We then train the model to make the two agree, using a ranking objective. The key design choice is a matched control. The same clinical outcome is also presented as if it came from a large randomized trial, which separates a model that tracks evidence strength from one that simply learns to hedge everywhere.

Core Claim

We claim that models over-generalize from single case reports when generating and are more conservative when validating. We also expect that surface hedging hides this. A generated recommendation can end with "further studies are needed" while its content still implies that the treatment works for the whole population. If this is right, two things follow. First, a consistency measure defined over the scope of a claim will expose failures that hallucination benchmarks and hedging-phrase counts miss. Second, aligning the generator's ranking of claims to the validator's ranking can fix them without making the model uniformly timid, provided training also includes cases where broad claims are warranted.

The second half of this claim is falsifiable in a specific way. The gap might run the other way, with a validator that accepts over-broad claims. Or scope might be consistent and wrong in both modes. Either result would redirect the work, as discussed below.

Positioning

Three lines of work motivate this proposal.

The generator–validator literature shows that generation and judgment come apart. Benchmarking and Improving Generator-Validator Consistency of Language Models found ChatGPT answering "15" to "7+8" while rejecting "7+8=15". It also showed that fine-tuning on self-consistent pairs raises agreement. RankAlign recast the gap as a disagreement in how the two modes rank candidate answers, and closed it with a pairwise ranking loss. Potemkin Understanding in Large Language Models showed that frontier models can define a concept yet misapply it, and often reject examples they themselves generated. All three study general-domain concepts. We carry the ranking view into a setting where the candidates are claims ordered by how far they generalize.

Two clinical papers sit closest to us, and they constrain the design. MHB uses PMC-Patients case reports with injected traps and an external judge model. It measures hallucination, not whether the model's own writing and judging agree. It also does not ask whether models respect the limits of a single case. Med-R² handles the evidence hierarchy outside the model: a reranker places trials above case reports during retrieval. Its ablations show this ranking matters most.

Med-R²'s approach has a blind spot that defines our regime. It reorders whatever evidence it retrieves. When a case report is the only evidence, as in rare diseases, there is nothing to reorder, and the model's internal handling decides the outcome.

Tension in the literature also shapes the method. RankAlign reports that aligning validator to generator works better than the reverse. TSQueryBench and Gaining Wisdom from Setbacks find the validator stronger in numeric-explanation and safety tasks. We therefore do not assume a direction. We measure which mode is closer to the expert scope label and align toward that mode.

Research Program

The scope ladder. For each case report drawn from PMC-Patients, we write a five-rung ladder of claims about its central outcome:

  • Rung 0 states that the outcome occurred in this patient.
  • Rung 1 states that it is possible in comparable patients.
  • Rung 2 states that it is likely in a subgroup defined by the patient's reported features.
  • Rung 3 states that it holds for the general population with the condition.
  • Rung 4 states that it should be standard of care.

The case reports in our file show what rung 2 must encode. The 92-year-old pancreaticoduodenectomy patient had an ECOG performance status of 1 (LPD in a 92-year-old). The HIV-positive gastrectomy patient had a CD4 count of 530/mm³ (gastrectomy with HIV). The sintilimab complete response came with grade 3 pneumonitis (PD-1 inhibitor CR). The 94% tumor reduction was followed by death two days after surgery (esophageal case).

For a single case report, the warranted maximum is rung 1. Rung 2 is acceptable only when labeled as a hypothesis.

The matched control. Each report gets a twin, written by a templated edit and checked by clinicians. The twin attributes the same outcome to a randomized trial of several hundred patients, so rungs 3 and possibly 4 become warranted. Clinicians annotate the warranted maximum rung for both versions on a validation subset. The ladder is built from model-agnostic templates plus clinician edits, so no evaluated model writes its own test items.

Measuring the generator. We measure the generator's scope in two ways.

  • Free generation. The model writes a clinical summary and recommendation. A held-out annotator model, calibrated against clinician labels on a subset, assigns the highest rung the text commits to. Hedging phrases are scored separately from the content-level commitment.
  • Log-probabilities. For open models, we score each rung's sentence as a continuation of "Based on this report, …". This mirrors RankAlign's generator scores and needs no annotator at all.

Measuring the validator. The validator's scope is the probability the model assigns to "True" when asked whether each rung is warranted by the report.

Metrics. Three numbers summarize consistency.

  • The scope gap is the generator's committed rung minus the highest rung the validator accepts.
  • The rank agreement is the Spearman correlation between generator and validator scores across the ladder.
  • The design sensitivity is how much each mode's scope rises from the case-report version to the trial version. Expert scope rises sharply.

The intervention. The method is a RankAlign-style pairwise loss over rungs, applied in whichever direction the measurements show is better grounded. It is trained on both case-report and trial versions, so the target is "track the design" rather than "always say rung 1".

Baselines

The do-nothing reference is each model evaluated as is. We use open models with accessible log-probabilities: Llama-3.1-8B and 70B and Qwen2.5-7B. We also measure frontier API models such as GPT-4o and Claude for measurement only. This establishes whether a gap exists at all. If the scope gap is near zero and both modes are near the expert label, the project stops at a negative measurement paper.

The instruction baseline prepends an explicit evidence-hierarchy instruction to both the writing and judging prompts. It is the cheapest fix and the most likely to be adopted in practice. We run it with the same prompts across all models. Beating it means a lower content-level scope gap, not merely more hedging phrases.

The self-reference baseline follows Do Before You Judge: the model writes its own recommendation, then judges each rung against it. That paper reports incorrect-answer detection F1 rising from 0.46–0.75 under standard chain-of-thought to as high as 0.97 with self-reference. It is the strongest inference-time competitor. It should also inherit the generator's over-reach, which is the error propagation that paper itself flags.

The two training baselines run on the same base models, the same ladder training split, and the same number of gradient steps and hyperparameter search budget.

  • Consistency fine-tuning from Benchmarking and Improving Generator-Validator Consistency. That paper reports GV-consistency rising from 60% to 94% on Alpaca-30B, against roughly 76% for GPT-4.
  • RankAlign with its default direction, validator aligned to generator. That paper reports a 31.8% average improvement in ρ-all and 94.2% ρ-all on hypernymy.

No result is reported on Lacuna for any of these methods on clinical scope reasoning or on PMC-Patients. Comparisons are therefore to our own reruns.

A retrieval-reranker baseline in the style of Med-R² is run in a mixed-evidence condition. There, a case report and a trial abstract with conflicting conclusions appear together. This tests whether reranking alone suffices when higher-tier evidence exists.

The key ablation trains our objective without the trial twins. We expect it to match our method on case reports. On the trial versions it should collapse design sensitivity, showing it learned blanket conservatism.

Beating the field means three things on held-out reports from unseen specialties:

  • a smaller content-level scope gap than every baseline;
  • design sensitivity closer to the clinicians' than the twin-free ablation achieves;
  • no loss on PubMedQA, where the file reports 68.1% for fine-tuned BioBERT against 78.0% human accuracy (PubMedQA).

Why not just…?

Why not just tell the model to respect the evidence hierarchy? This is the most serious threat, and it may fully fix the validator. Our hypothesis is that it fixes the wording of generation but not its content: models will append caveats while still recommending the treatment broadly. Two pieces of evidence point that way. Potemkin models recite definitions they then misapply. MHB's physician-written rubrics did not prevent hallucination rates of 29.1% for Claude-4-Sonnet and 59.0% for Llama-4-Scout. Scoring content commitment separately from hedge phrases settles the question directly. If the instruction alone brings content-level scope to the clinicians' label on both versions, we report that, and the training component is unnecessary.

Why not just use an evidence-hierarchy reranker as Med-R² does? A reranker can only reorder what is retrieved. In the case-report-only regime, which is common for rare conditions, it has nothing to reorder. Our mixed-evidence condition still tests it where it applies. If it matches our method there, the honest conclusion is that internal alignment matters mainly when only a single case is available.

Why not just have the model generate first and then judge (self-reference)? Self-reference makes the judge agree with the writer. If the writer over-generalizes, this makes things worse, not better. The measured direction of the gap decides this. If the generator turns out to be the better-grounded mode, self-reference becomes the natural solution and our contribution narrows to the benchmark and the twin-based diagnostic.

Decisive Checks

The first check is whether a content-level scope gap exists in untuned models, and in which direction. The direction is read from which mode sits closer to the clinician label on the validation subset. The second is whether the instruction baseline closes it.

The third check is the twin-free ablation. It shows whether gains come from tracking study design or from blanket hedging. If our method lowers scope on case reports but also lowers it on trial versions, it has failed regardless of headline gap reduction.

Finally, we report the MHB hallucination rate before and after training. This side check guards against the model getting better at stating limitations while getting worse at reading the case.