TL;DR
CaseScope is a new benchmark designed to evaluate how Multimodal Large Language Models (MLLMs) handle "case-level" pathology—the process of synthesizing a diagnosis from multiple images of a single lesion at different magnifications. The study reveals that current models are surprisingly fragile: they are highly sensitive to the order of multiple-choice options, often ignore visual evidence in favor of text-based shortcuts, and struggle to distinguish between classic "look-alike" diseases.
What Problem Does This Paper Address?
In the real world, a surgical pathologist doesn't just look at one picture to make a diagnosis. They follow a multi-step workflow: scanning a slide at low power to see the overall "architecture" (how the tissue is organized), then zooming in to high power to examine "cytology" (individual cell details like nuclear size or mitoses).
Existing AI benchmarks for pathology have a major gap. Most focus on single-image tasks, which are too simple, or whole-slide image (WSI) tasks, which are so computationally heavy that they mostly test specialized slide encoders rather than the reasoning capabilities of MLLMs. CaseScope fills this middle ground by providing curated sets of 2–8 fields per lesion, forcing models to integrate evidence across different views and scales—just like a human resident in training.
Methods
The authors introduced a rigorous evaluation framework centered on two primary data sources and six distinct tasks. Key methodological innovations include:
- Circular Evaluation: To expose "position bias," every multiple-choice question is asked ten times, with the options rotated each time. A model only gets credit if it picks the correct answer in every rotation.
- Evidence Ablation: The researchers systematically removed images or text hints (like the specimen site) to quantify exactly how much the models rely on visual vs. textual information.
- Ontology-Guided Distractors: Instead of random wrong answers, the benchmark uses a WHO-based ontology to select "hard negatives"—classic look-alike diseases that are known to confuse human pathologists.
- Hierarchical Scoring: For open-ended diagnosis, models receive partial credit for identifying the correct "family" of diseases, even if they miss the exact sub-type.
Datasets
CaseScope consists of two complementary sources covering 843 unique disease entities across 18 organ systems:
- CaseScope-Teaching (436 cases): Derived from a residency training library. These cases feature 2,448 H&E fields (median 5 per case) ordered from low to high magnification. They are pathologist-verified and represent the core curriculum for medical residents.
- CaseScope-Atlas (800 cases): Sourced from 88 published pathology atlases. These provide a broader range of rare diseases and test if model performance carries over to different image sources.
Key Insights and Ideas
- Diagnosis is Unstable: Even the best models are "fragile." A model might correctly diagnose a case in one rotation but fail if the same options are presented in a different order. Only 17.9% of teaching cases were correctly diagnosed by the best open model across all rotations.
- The "Text Shortcut" Trap: Open-source models often ignore the additional images provided. In many cases, removing the one-line text hint about the specimen site (e.g., "skin" or "lung") hurt performance more than removing almost all the images.
- Verification vs. Hallucination: Models often struggle to verify the actual morphological findings (like "nuclear pleomorphism") that justify a diagnosis. They might guess the right disease name based on the specimen site but fail to "see" the visual evidence in the images.
- Look-alike Attraction: When models fail, they tend to pick the classic "look-alike" distractor, suggesting they are learning superficial correlations rather than robust diagnostic criteria.
Key Results
The study evaluated 28 open-source and 4 proprietary MLLMs.
- Proprietary Lead: Proprietary models (like Claude 3.5 Sonnet and GPT-4o variants) significantly outperformed open models, reaching up to 61.7% consistency on the teaching cases, compared to a maximum of 17.9% for open models.
- Finding Verification: Most open models performed barely above chance (median 62.0% vs. 50% chance) when asked to judge if specific morphological statements were true or false based on the images.
- Integration Failure: For open models, keeping only the single highest-power image resulted in nearly the same accuracy as providing the full multi-field case. This indicates that current open MLLMs are not yet effectively "integrating" evidence across multiple scales.
Limitations and Future Work
The authors acknowledge several limitations:
- Fixed Field Selection: The benchmark uses fields already selected by experts. In practice, a pathologist must find these fields themselves on a massive gigapixel slide.
- Stain Variety: The current version is limited to H&E (Hematoxylin and Eosin) stains, excluding immunohistochemistry (IHC) or special stains often needed for definitive diagnosis.
- Future Directions: The authors suggest that future work should focus on "agentic" models that can navigate whole slides themselves, using CaseScope as a test of their reasoning once they've arrived at the relevant fields.
Summary updated