Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models

Katherine Elfer, Emma Gardecki, Víctor García, Amy Ly, Evangelos Hytopoulos, Si Wen, Matthew G. Hanna, Dieter Peeters, Joel Saltz, Anna Ehinger, Sarah Dudgeon, Xiaoxian Li, Kim Blenman, Weijie Chen, Ursula Green, Ryan Birmingham, Tony Pan, Jochen K. Lennerz, Roberto Salgado, Brandon D. GallasModern 2024 · DOI

TL;DR

This paper introduces CLEARR-AI, a comprehensive framework and checklist designed to standardize the reporting of how annotations are collected and evaluated for AI models in medical imaging. By operationalizing existing best practices, CLEARR-AI aims to improve the transparency, reproducibility, and quality of annotation datasets, which are crucial for developing reliable AI diagnostic tools.

What Problem Does This Paper Address?

The rapid advancement of Artificial Intelligence (AI) in medical image analysis, particularly in areas like digital pathology, hinges on the quality and transparency of the data used to train and validate these models. However, a significant challenge lies in the inconsistent and often insufficient reporting of how annotations (e.g., marking regions of interest, classifying abnormalities) are created and assessed. This lack of standardization makes it difficult to reproduce studies, compare models, and ensure the reliability of AI-driven diagnostic tools. Existing reporting guidelines for clinical trials (CONSORT, SPIRIT) and diagnostic accuracy studies (STARD, TRIPOD) are being extended to address AI, but there's a specific need for a framework focused on the human annotation process itself, which is a critical component of medical AI development.

Methods

The authors propose and demonstrate the utility of the CLEARR-AI (Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence) framework. This framework consolidates and operationalizes a previously proposed 11-step workflow for annotation data set construction into 7 essential components. These components cover:

  1. Objectives: Defining the AI/ML use case, target population, and annotation degree.
  2. Data Dictionary: Creating a comprehensive guide for metadata and annotation features, including definitions, examples, and training materials.
  3. Study Design: Specifying workload distribution, time limits, and adjudication methods for annotators.
  4. Annotation Methods: Detailing the tools and techniques used for annotation.
  5. Image Curation: Describing sampling strategies and the rationale for including images of varying quality.
  6. Annotators: Documenting recruitment methods, qualifications, and experience.
  7. Quality Review: Establishing a process for ongoing assessment and final review of the annotation process and data.

The paper applies this framework to a pilot study of the High-Throughput Truthing (HTT) project, which focuses on annotating stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsies.

Datasets

The paper primarily discusses the use of data from the High-Throughput Truthing (HTT) project's pilot study. This dataset consists of H&E-stained breast cancer biopsy slides, specifically focusing on triple-negative breast cancer (TNBC). The annotations collected include:

  • ROI Type: Whether a region of interest (ROI) is evaluable for sTIL evaluation.
  • % Tumor-Associated Stroma: A quantitative measure of the stromal area within the ROI.
  • % sTIL Density: A quantitative measure of the density of stromal tumor-infiltrating lymphocytes within the tumor-associated stroma.

The paper also references the work of Wahab et al. as a foundational dataset for digital pathology annotations. The HTT project's data, including scripts and functions, is made publicly available on GitHub.

Key Insights and Ideas

  • Annotation is a Critical Bottleneck: The paper strongly emphasizes that the quality and reproducibility of AI models are directly tied to the quality and transparency of the annotation process. Inadequate reporting here can undermine even the most sophisticated AI algorithms.
  • Standardization is Paramount: The CLEARR-AI framework provides a much-needed standardized approach to reporting annotations, moving beyond ad-hoc methods and fostering trust in AI-driven medical insights.
  • Beyond "Ground Truth": The framework acknowledges that "ground truth" is often derived from human interpretation, which inherently has variability. CLEARR-AI aims to document and manage this variability through detailed reporting of annotator qualifications and quality review processes.
  • Iterative Improvement: The application of CLEARR-AI to the HTT pilot study highlighted areas for improvement, such as enhancing annotator training and workload distribution, demonstrating the framework's practical value in refining annotation protocols for subsequent studies.
  • Broad Applicability: While demonstrated with digital pathology, the CLEARR-AI framework is designed to be generalizable to any reader-interpreted medical imaging annotation effort.

Key Results

The paper demonstrates the utility of the CLEARR-AI framework by applying it to the HTT pilot study. This application allowed the researchers to:

  • Document the annotation process: All 7 components of the CLEARR-AI framework were addressed, providing a comprehensive overview of the pilot study's annotation efforts.
  • Identify shortcomings: The review revealed limitations in the pilot study, such as an unbalanced distribution of pathologists per ROI and a lack of enforced training completion.
  • Inform improvements for the pivotal study: Based on the pilot study's review, significant changes were implemented for the pivotal study, including enhanced training modules (CME course, interactive tests), improved data collection platforms, and more controlled workload distribution.
  • Facilitate understanding: The framework's application helped others understand the key elements of the pilot study, promoting transparency and reproducibility.

Limitations and Future Work

  • Limited Scope of Demonstration: The authors acknowledge that the CLEARR-AI framework was demonstrated using only one annotation study (HTT) within a single medical imaging domain (digital pathology). They note that the HTT project team was already highly familiar with data collection and reporting, which may not reflect the challenges faced by less experienced teams.
  • Not a Rigorous Consensus Process: The framework is not the result of a formal, broad consensus process involving all stakeholders in the AI/ML community, unlike established standards like STARD or TRIPOD.
  • Future Work: The authors suggest that future adoption and feedback from a wider range of data set developers and users in the medical imaging community will lead to further improvements in reporting consistency and clarity. They also anticipate that challenge platforms (e.g., Grand Challenges, Kaggle) may adopt similar reporting standards. The paper also highlights the ongoing work to validate that annotations collected with digital images are equivalent to those collected on microscopes.

Summary updated

Abstract

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).
BibTeX
@inproceedings{elfer2024reproducible,
  author = {Katherine Elfer and Emma Gardecki and Víctor García and Amy Ly and Evangelos Hytopoulos and Si Wen and Matthew G. Hanna and Dieter Peeters and Joel Saltz and Anna Ehinger and Sarah Dudgeon and Xiaoxian Li and Kim Blenman and Weijie Chen and Ursula Green and Ryan Birmingham and Tony Pan and Jochen K. Lennerz and Roberto Salgado and Brandon D. Gallas},
  title = {{Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models}},
  booktitle = {Modern},
  year = {2024},
  doi = {10.1016/j.modpat.2024.100439},
}
Figure