CaseScope: A Multi-Field Benchmark for Case-Level Pathology Diagnosis with Multimodal LLMs

Wenchuan Zhang, Chengyang Zhang, Fengling Li, Ting Yin, Chen Shu, Jie Chen, Shuwan Zhang, Xiaoyan Wu, Yuhao Yi, Jiancheng Lv, Hong BuarXiv 2026 · arXiv

TL;DR

CaseScope is a new benchmark designed to evaluate how Multimodal Large Language Models (MLLMs) handle "case-level" pathology—the process of synthesizing a diagnosis from multiple images of a single lesion at different magnifications. The study reveals that current models are surprisingly fragile: they are highly sensitive to the order of multiple-choice options, often ignore visual evidence in favor of text-based shortcuts, and struggle to distinguish between classic "look-alike" diseases.

What Problem Does This Paper Address?

In the real world, a surgical pathologist doesn't just look at one picture to make a diagnosis. They follow a multi-step workflow: scanning a slide at low power to see the overall "architecture" (how the tissue is organized), then zooming in to high power to examine "cytology" (individual cell details like nuclear size or mitoses).

Existing AI benchmarks for pathology have a major gap. Most focus on single-image tasks, which are too simple, or whole-slide image (WSI) tasks, which are so computationally heavy that they mostly test specialized slide encoders rather than the reasoning capabilities of MLLMs. CaseScope fills this middle ground by providing curated sets of 2–8 fields per lesion, forcing models to integrate evidence across different views and scales—just like a human resident in training.

CaseScope overview showing how it bridges the gap between single-image and whole-slide benchmarks.

Methods

The authors introduced a rigorous evaluation framework centered on two primary data sources and six distinct tasks. Key methodological innovations include:

  • Circular Evaluation: To expose "position bias," every multiple-choice question is asked ten times, with the options rotated each time. A model only gets credit if it picks the correct answer in every rotation.
  • Evidence Ablation: The researchers systematically removed images or text hints (like the specimen site) to quantify exactly how much the models rely on visual vs. textual information.
  • Ontology-Guided Distractors: Instead of random wrong answers, the benchmark uses a WHO-based ontology to select "hard negatives"—classic look-alike diseases that are known to confuse human pathologists.
  • Hierarchical Scoring: For open-ended diagnosis, models receive partial credit for identifying the correct "family" of diseases, even if they miss the exact sub-type.

Datasets

CaseScope consists of two complementary sources covering 843 unique disease entities across 18 organ systems:

  1. CaseScope-Teaching (436 cases): Derived from a residency training library. These cases feature 2,448 H&E fields (median 5 per case) ordered from low to high magnification. They are pathologist-verified and represent the core curriculum for medical residents.
  2. CaseScope-Atlas (800 cases): Sourced from 88 published pathology atlases. These provide a broader range of rare diseases and test if model performance carries over to different image sources.
Composition of the CaseScope datasets by organ system and number of fields per case.

Key Insights and Ideas

  • Diagnosis is Unstable: Even the best models are "fragile." A model might correctly diagnose a case in one rotation but fail if the same options are presented in a different order. Only 17.9% of teaching cases were correctly diagnosed by the best open model across all rotations.
  • The "Text Shortcut" Trap: Open-source models often ignore the additional images provided. In many cases, removing the one-line text hint about the specimen site (e.g., "skin" or "lung") hurt performance more than removing almost all the images.
  • Verification vs. Hallucination: Models often struggle to verify the actual morphological findings (like "nuclear pleomorphism") that justify a diagnosis. They might guess the right disease name based on the specimen site but fail to "see" the visual evidence in the images.
  • Look-alike Attraction: When models fail, they tend to pick the classic "look-alike" distractor, suggesting they are learning superficial correlations rather than robust diagnostic criteria.

Key Results

The study evaluated 28 open-source and 4 proprietary MLLMs.

  • Proprietary Lead: Proprietary models (like Claude 3.5 Sonnet and GPT-4o variants) significantly outperformed open models, reaching up to 61.7% consistency on the teaching cases, compared to a maximum of 17.9% for open models.
  • Finding Verification: Most open models performed barely above chance (median 62.0% vs. 50% chance) when asked to judge if specific morphological statements were true or false based on the images.
  • Integration Failure: For open models, keeping only the single highest-power image resulted in nearly the same accuracy as providing the full multi-field case. This indicates that current open MLLMs are not yet effectively "integrating" evidence across multiple scales.
Ablation results showing how removing the specimen site hint significantly drops performance for most models.

Limitations and Future Work

The authors acknowledge several limitations:

  • Fixed Field Selection: The benchmark uses fields already selected by experts. In practice, a pathologist must find these fields themselves on a massive gigapixel slide.
  • Stain Variety: The current version is limited to H&E (Hematoxylin and Eosin) stains, excluding immunohistochemistry (IHC) or special stains often needed for definitive diagnosis.
  • Future Directions: The authors suggest that future work should focus on "agentic" models that can navigate whole slides themselves, using CaseScope as a test of their reasoning once they've arrived at the relevant fields.

Summary updated

Abstract

Pathologists rarely diagnose from a single image: they scan a section at low power for architecture, zoom into selected fields for cytology, and integrate the evidence across views and scales. Pathology benchmarks for multimodal large language models (MLLMs), however, mostly ask about one image or require whole-slide pipelines, and the few that show several fields draw nested regions from cancer slides of one public cohort. We introduce CaseScope, a benchmark of diagnosis from several curated fields of one lesion: CaseScope-Teaching, 436 cases from a residency-training image library, reconstructed, de-identified, harmonised against a WHO-based ontology and re-examined by a pathologist wherever an automated pre-review raised a doubt, and CaseScope-Atlas, 800 cases from 88 published atlases (387 of them figure series of one disease), labelled from the captions by two LLMs that must agree. They cover 843 entities across 18 organ systems and pose 4,679 items in six tasks, from open-ended and ten-way diagnosis with ontology-guided hard negatives under circular evaluation to the verification and location of morphological findings. Across twenty-eight open and four proprietary MLLMs, diagnoses are unstable -- the best open and proprietary models are correct under all ten option rotations on 17.9% and 61.7% of the teaching cases -- and wrong answers favour the classic look-alike; open models make little use of the additional fields on teaching cases, where the one-line specimen site matters more, although on atlas cases with three or more fields the fields count about as much as the site; and most open models verify findings barely above chance (median 62.0% vs. 50%, best 81.5%), gaining a median +3.4 points from images; proprietary models reach 83.2-93.2%. Rankings broadly replicate between the sources. The evaluation code is available at https://github.com/Wenchuan-Zhang/CaseScope.
BibTeX
@misc{zhang2026casescope,
  author = {Wenchuan Zhang and Chengyang Zhang and Fengling Li and Ting Yin and Chen Shu and Jie Chen and Shuwan Zhang and Xiaoyan Wu and Zhang Zhang and Yuhao Yi and Jiancheng Lv and Hong Bu},
  title = {{CaseScope: A Multi-Field Benchmark for Case-Level Pathology Diagnosis with Multimodal LLMs}},
  year = {2026},
  eprint = {2611.15325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15325},
}
Figure