Pathology’s Last Exam: Stress-Testing Diagnostic Reasoning and Safety in Large Language Models

Nic G. Reitsam, Marco Gustav, Moritz Jesinghaus, Bruno Märkl, Sebastian Foersch, Jakob Nikolas KathermedRxiv 2025 · DOI

Abstract

Large language models (LLMs) are evolving into diagnostic co-pilots, yet current benchmarks fail to test the integrated, stepwise reasoning required in diagnostic pathology. Here, we present Pathology’s Last Exam (PLE), a curated, highly detailed, text-based benchmark of 100 complex cases spanning organ systems, enriched for rare/challenging entities, plus 20 adversarial cases designed to stress-test model safety. Each case provides structured blocks (Primary, Clinical, Histopathology, IHC/Special Stains, Molecular Pathology) with stepwise information release mirroring real sign-out. We evaluated five LLMs (one proprietary, four open-source) across different stages. While the best model (GPT-5) achieved 70% accuracy on full evidence, performance on safety tests was alarming. Models frequently failed to detect biological contradictions, confidently diagnosing nonsensical “mix-up” cases rather than refusing them. This reveals a critical safety gap: high diagnostic capability is currently coupled with a dangerous inability to recognize impossible clinical scenarios. PLE provides a framework to measure and mitigate these risks before clinical deployment, as well as a foundation for developing multimodal evaluation protocols that can be extended to vision-language models and autonomous diagnostic agents in the future.

Introduction

In the real world, a pathologist does not simply look at a single slide and select an answer from a list. Diagnosis is a sequential, integrative process: it begins with basic morphology, incorporates immunohistochemistry (IHC) stains, and finally synthesizes molecular data and clinical history. Current AI benchmarks often fail to capture this complexity, focusing on common diseases and multiple-choice formats that may inflate a model's perceived competence—a phenomenon the authors describe as an "illusion of readiness."

To address this, researchers have introduced Pathology’s Last Exam (PLE), a rigorous new benchmark designed to move beyond simple classification and test the "stepwise" reasoning of Large Language Models (LLMs) in complex pathology.

The PLE Framework

The study introduces a dataset designed to push models to their limits through two primary components:

  • 100 Complex Cases: Highly detailed, text-based vignettes spanning various organ systems, specifically enriched for rare, emerging, and "borderline" entities that are often underrepresented in standard datasets like The Cancer Genome Atlas (TCGA).
  • 20 Safety/Adversarial Cases:
    • Biologically Nonsensical: 10 cases with "lineage clashes" where markers are provided that cannot biologically coexist in the same cell type.
  • "Mix-up" Cases: 10 cases where data from different patients were randomly recombined to simulate laboratory errors.

Stepwise Evaluation Methodology

The researchers developed a staged evaluation framework mirroring the clinical "sign-out" workflow. Information was released to five LLMs (GPT-5, GPT-OSS-120B, Llama-4, Qwen3, and MedGemma) in four progressively detailed stages:

  1. PH: Primary Organ + Histopathology (Morphology only).
  2. PHS: PH + Special Stains/IHC.
  3. PHSM: PHS + Molecular Pathology.
  4. PHSMC: PHSM + Clinical Summary (Full evidence).

Models were instructed to use an "insufficiency token" if the data was too contradictory or sparse to reach a definitive diagnosis. Their outputs were then blindly rated by human pathologists as Incorrect, Partly Correct, or Correct.

Key Results and Performance Gaps

Diagnostic Accuracy

GPT-5 led the evaluation, achieving 70% accuracy on the full evidence (PHSMC) stage. However, a significant gap remains between proprietary and open-source models; MedGemma, for instance, scored as low as 17% on the same tasks. The study confirmed that accuracy improves significantly as more "layers" of data (IHC, molecular) are added, proving that LLMs can integrate multimodal text data effectively when it is consistent.

The Safety Failure

The most critical finding was the dangerous decoupling between diagnostic capability and safety. No model achieved universal refusal on adversarial cases. Even GPT-5, the top performer, only refused 70% of adversarial cases and 50% of "mix-up" cases. Other models frequently issued confident, "overclaimed" diagnoses for biologically impossible scenarios.

Discussion: The Sycophancy Trap

The study highlights a "sycophancy trap" in current LLMs: models prioritize being "helpful" and compliant over being logically consistent. When presented with nonsensical or contradictory data, models would rather invent a diagnosis than admit the data is impossible. This reveals a critical safety gap where high diagnostic skill is paired with a dangerous inability to recognize clinical errors or impossible scenarios.

Limitations and Future Work

The authors acknowledge that the benchmark is currently text-only, whereas real pathology relies heavily on Whole-Slide Images (WSIs). Additionally, the cases reflect a Western bias, primarily drawn from European and US pathology journals.

Future work aims to expand PLE into a multimodal benchmark including images and raw molecular data. The team also suggests exploring "LLM-as-judge" architectures to automate the grading of complex, free-text diagnostic reasonings, providing a foundation for evaluating autonomous diagnostic agents in the future.

Summary updated

Abstract

Abstract Large language models (LLMs) are evolving into diagnostic co-pilots, yet current benchmarks fail to test the integrated, stepwise reasoning required in diagnostic pathology. Here, we present Pathology’s Last Exam (PLE) , a curated, highly detailed, text-based benchmark of 100 complex cases spanning organ systems, enriched for rare/challenging entities, plus 20 adversarial cases designed to stress-test model safety. Each case provides structured blocks (Primary, Clinical, Histopathology, IHC/Special Stains, Molecular Pathology) with stepwise information release mirroring real sign-out. We evaluated five LLMs ( one proprietary, four open-source) across different stages. While the best model (GPT-5) achieved 70% accuracy on full evidence, performance on safety tests was alarming. Models frequently failed to detect biological contradictions, confidently diagnosing nonsensical “mix-up” cases rather than refusing them. This reveals a critical safety gap: high diagnostic capability is currently coupled with a dangerous inability to recognize impossible clinical scenarios. PLE provides a framework to measure and mitigate these risks before clinical deployment, as well as a foundation for developing multimodal evaluation protocols that can be extended to vision-language models and autonomous diagnostic agents in the future.
BibTeX
@misc{reitsam2025pathologys,
  author = {Nic G. Reitsam and Marco Gustav and Moritz Jesinghaus and Bruno Märkl and Sebastian Foersch and Jakob Nikolas Kather},
  title = {{Pathology’s Last Exam: Stress-Testing Diagnostic Reasoning and Safety in Large Language Models}},
  year = {2025},
  doi = {10.64898/2025.12.11.25342081},
  howpublished = {medRxiv},
}
Figure