A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study

Sarah Dudgeon, Si Wen, Matthew G. Hanna, Rajarsi Gupta, Mohamed Amgad, Manasi Sheth, Hetal D. Marble, Richard Huang, Markus D. Herrmann, Clifford H. Szu, Darick Tong, Bruce Werness, Evan Szu, Denis Larsimont, Anant Madabhushi, Evangelos Hytopoulos, Weijie Chen, Vikram Singh, Steven N. Hart, Joel Saltz, Roberto Salgado, Brandon D. GallasJournal of Pathology Informatics 2021 · DOI

Abstract

We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.

Introduction: The "Ground Truth" Problem

In medical artificial intelligence (AI), validating algorithms for clinical use remains a significant bottleneck. In digital pathology, the reference standard—or "ground truth"—is typically the opinion of a pathologist. However, human experts are "noisy" sensors; different pathologists often disagree on the same image. Furthermore, most validation data is siloed within private institutions, preventing regulators like the FDA from performing standardized comparisons on an apples-to-apples basis.

The "High-Throughput Truthing" (HTT) project addresses this by creating a massive, pathologist-annotated dataset designed to move AI from research curiosities to FDA-qualified Medical Device Development Tools (MDDT). This paper describes a pilot study focused on estimating the density of stromal tumor-infiltrating lymphocytes (sTILs), a critical prognostic biomarker in breast cancer.

Methods and Study Design

The HTT project utilizes a specialized workflow to collect annotations at scale. In the pilot, 19 pathologists (both board-certified and residents) evaluated 64 digitized glass slides of H&E-stained invasive ductal carcinoma. Key methodological features include:

  • Multi-Platform Evaluation: Annotations were collected via web-based platforms (PathPresenter, caMicroscope) and a specialized hardware-software system (eeDAP) that interfaces traditional microscopes with digital images.
  • Structured ROI Selection: A lead pathologist pre-selected 10 Regions of Interest (ROIs) per slide to represent diverse morphologies, such as invasive margins and necrosis, rather than relying on whole-slide guessing.
  • Statistical Modeling: The study employs Root Mean Squared Error (RMSE) and Multi-Reader Multi-Case (MRMC) methods. These models account for the correlation between ROIs from the same patient and treat pathologist variability as a component of the "truth" rather than simple error.

Validation Frameworks and Regulatory Pathfinding

A core objective of the HTT project is to establish a "pre-qualified" sandbox for AI developers. By pursuing MDDT qualification, the authors are defining the statistical and procedural standards required for regulatory acceptance, potentially saving years of regulatory back-and-forth for developers.

Overview of the HTT validation frameworks, illustrating the progression from analytical validation to clinical validation (independent crossover and sequential designs) within the MDDT process.

As shown in the figure above, the project outlines three primary validation paths:

  1. Analytical Validation: Directly comparing AI output to the established reference standard.
  2. Clinical Validation (Independent Crossover): Comparing clinical reads performed without AI aid against those performed with AI aid using an independent crossover design.
  3. Clinical Validation (Sequential Study Design): A sequential approach to evaluating the impact of AI aid on clinical decision-making.

Key Insights and Results

The pilot study generated over 1,600 evaluations, confirming significant inter-pathologist variability. Key findings include:

  • Pathologists as "Noisy Realizations": Rather than seeking a single "perfect" expert, the study suggests that statistical averaging of multiple reader scores provides a more stable and reliable reference standard.
  • The "IID" Fallacy: Standard AI validation often assumes data points are Independent and Identically Distributed (IID). This study highlights that ROIs from the same slide are highly correlated, necessitating nested statistical models to avoid overestimating performance.
  • Data Distribution: The pilot found a high prevalence of low sTIL infiltration. This underscores the need for "data curation" (stratification) in pivotal studies to ensure the AI is validated across the full range of lymphocyte densities.

Conclusion and Future Work

The HTT project is transitioning from a pilot to a pivotal study. Future work will involve expanding the dataset to include slides from multiple institutions to account for pre-analytic variability (e.g., tissue preparation and staining). Additionally, the team will implement more rigorous pathologist training and proficiency tests to reduce noise in the reference standard. Ultimately, the goal is to provide a public, regulatory-grade dataset that accelerates the clinical adoption of AI in pathology.

Summary updated

Abstract

We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.
BibTeX
@article{dudgeon2021pathologist,
  author = {Sarah Dudgeon and Si Wen and Matthew G. Hanna and Rajarsi Gupta and Mohamed Amgad and Manasi Sheth and Hetal D. Marble and Richard Huang and Markus D. Herrmann and Clifford H. Szu and Darick Tong and Bruce Werness and Evan Szu and Denis Larsimont and Anant Madabhushi and Evangelos Hytopoulos and Weijie Chen and Vikram Singh and Steven N. Hart and Joel Saltz and Roberto Salgado and Brandon D. Gallas and Brandon D. Gallas},
  title = {{A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study}},
  journal = {Journal of Pathology Informatics},
  year = {2021},
  doi = {10.4103/jpi.jpi\_83\_20},
}
Figure