Computer-Assisted Mitotic Count Using a Deep Learning-based Algorithm Improves Inter-Observer Reproducibility and Accuracy in canine cutaneous mast cell tumors

Christof Bertram, Marc Aubreville, Taryn Donovan, Alexander Bartel, Frauke Wilm, Christian Marzahl, Charles‐Antoine Assenmacher, Kathrin Becker, Mark K. Bennett, Sarah Corner, Brieuc Cossic, Daniela Denk, Martina Dettwiler, Beatriz Garcia Gonzalez, Corinne Gurtner, Ann-Kathrin Haverkamp, Annabelle Heier, Annika Lehmbecker, Sophie Merz, Erica L. Noland, Stephanie Plog, Anja Schmidt, Franziska Sebastian, Dodd G. Sledge, Rebecca C. Smedley, Marco Tecilla, Tuddow Thaiwong, Andrea Fuchs‐Baumgartinger, Don J. Meuten, Katharina Breininger, Matti Kiupel, Andreas Maier, Robert KlopfleischbioRxiv 2021 · DOI

TL;DR

This paper introduces a deep learning-assisted system to improve the accuracy and consistency of counting mitotic figures (MFs) in canine mast cell tumors. By providing pathologists with algorithmic preselection of regions of interest and highlighting potential MFs, the system significantly boosts inter-observer agreement and individual MF detection performance.

What Problem Does This Paper Address?

Accurately counting mitotic figures (MFs) in histological tumor samples is crucial for predicting patient prognosis, especially in canine cutaneous mast cell tumors (ccMCTs). However, this process is notoriously prone to significant variability between pathologists (inter-observer discrepancies) and even by the same pathologist at different times (intra-observer discrepancies). This inconsistency stems from challenges in identifying the correct region within the tumor to count (the "mitotic hotspot") and in accurately distinguishing true MFs from similar-looking cellular structures (imposters). These issues hinder reliable prognostication and standardization of diagnostic practices.

Methods

The study employed a multi-stage approach to evaluate the impact of computer assistance on MF counting.

  • Stage 1 (Unaided): Pathologists manually identified regions of interest (MC-ROIs) and counted MFs using their standard methods on whole slide images (WSIs).
  • Stage 2 (Partial Computer Assistance): A deep learning algorithm preselected the MC-ROI, highlighting areas with the highest predicted mitotic activity. Pathologists then counted MFs within this preselected region.
  • Stage 3 (Full Computer Assistance): In addition to the preselected MC-ROI, the algorithm also visualized individual MF candidates and their confidence scores as an overlay on the WSI. Pathologists reviewed these suggestions and made their final counts.

A key methodological innovation was the development of a "ground truth" dataset using immunohistochemistry for phosphohistone H3 (pHH3). This technique specifically labels cells in the mitotic phase, providing a highly accurate reference for MF identification, which was then compared against both pathologist and algorithmic counts.

Datasets

  • Training Data: The deep learning algorithm was trained on a dataset of 44,880 MF annotations from 32 ccMCT WSIs.
  • Evaluation Data: The study evaluated 50 ccMCT cases. A subset of 40 cases was used to generate the pHH3-assisted ground truth dataset.

Key Insights and Ideas

  • AI as a "Second Pair of Eyes": The study strongly suggests that AI isn't meant to replace pathologists but to augment their capabilities, particularly in tedious and subjective tasks like MF counting.
  • Hotspot Identification is Key: The difficulty in finding the "mitotic hotspot" is a major source of variability. Algorithmic preselection of these hotspots proved to be a highly effective intervention.
  • Confidence Scores Matter: While not universally embraced, the display of algorithmic confidence scores for MF candidates did influence pathologist decisions, especially when they were uncertain.
  • Ground Truth is Crucial for Validation: The use of pHH3-assisted immunohistochemistry provided a robust, objective benchmark that was essential for accurately assessing both algorithmic and human performance.

Key Results

  • Improved Inter-Observer Agreement: Computer assistance significantly increased the consistency of MF counts between pathologists. The inter-observer correlation coefficient (ICC) rose from 0.70 (unaided) to 0.92 (full computer assistance).
  • Enhanced Individual MF Detection: When using full computer assistance, pathologists' performance in detecting individual MFs improved, with an F1-score increasing from 0.68 to 0.79. This was accompanied by a 38% reduction in false negatives.
  • Higher Mitotic Counts with Assistance: Computer-assisted methods consistently yielded higher mitotic counts compared to unaided manual counts, likely due to improved detection of MFs and hotspots.
  • Accurate Prognostic Classification: Computer assistance improved the accuracy of classifying cases into prognostic categories based on a standard cut-off value (MC ≥ 5), with accuracy increasing from 75.8% (stage 1) to 91.7% (stage 3) compared to the ground truth.
  • Algorithmic Preselection Superiority: Algorithmic MC-ROI preselection was found to be superior to manual selection in identifying mitotic hotspots, with 79% of algorithmically selected ROIs showing higher MCs than manually selected ones.

Limitations and Future Work

  • Case Selection Bias: The study's case selection strategy, focusing on cases with high mitotic density, may not fully represent routine diagnostic scenarios. Future studies should verify results on a broader range of cases, especially those with very low MCs.
  • Algorithmic Errors: Despite high accuracy, the algorithm did produce some incorrect MF predictions and inappropriate MC-ROI preselections, highlighting the need for expert review.
  • Domain Shift: The algorithm's performance on images from different scanners or with varying staining characteristics (domain shift) was not evaluated and requires further validation.
  • Prognostic Value: The study focused on MC accuracy and reproducibility, not directly on prognostic outcomes. Future research should investigate how computer-assisted MCs impact patient prognosis.
  • MC-ROI Shape: While the rectangular shape was generally accepted, its suitability for all tumor types with variable cellular density or non-neoplastic tissue warrants further investigation.
  • Expert Review Level: Determining the optimal degree of expert review required for computer-assisted MCs remains an open question.
  • Bias from Confidence Scores: The potential for algorithmic confidence scores to introduce bias (positive or negative) in pathologist decision-making needs further study.

Summary updated

Abstract

Abstract The mitotic count (MC) is an important histological parameter for prognostication of malignant neoplasms. However, it has inter- and intra-observer discrepancies due to difficulties in selecting the region of interest (MC-ROI) and in identifying/classifying mitotic figures (MFs). Recent progress in the field of artificial intelligence has allowed the development of high-performance algorithms that may improve standardization of the MC. As algorithmic predictions are not flawless, the computer-assisted review by pathologists may ensure reliability. In the present study we have compared partial (MC-ROI preselection) and full (additional visualization of MF candidate proposal and display of algorithmic confidence values) computer-assisted MC analysis to the routine (unaided) MC analysis by 23 pathologists for whole slide images of 50 canine cutaneous mast cell tumors (ccMCTs). Algorithmic predictions aimed to assist pathologists in detecting mitotic hotspot locations, reducing omission of MF and improving classification against imposters. The inter-observer consistency for the MC significantly increased with computer assistance (interobserver correlation coefficient, ICC = 0.92) compared to the unaided approach (ICC = 0.70). Classification into prognostic stratifications had a higher accuracy with computer assistance. The algorithmically preselected MC-ROIs had a consistently higher MCs than the manually selected MC-ROIs. Compared to a ground truth (developed with immunohistochemistry for phosphohistone H3), pathologist performance in detecting individual MF was augmented when using computer assistance (F1-score of 0.68 increased to 0.79) with a reduction in false negatives by 38%. The results of this study prove that computer assistance may lead to a more reproducible and accurate MCs in ccMCTs.
BibTeX
@misc{bertram2021computer,
  author = {Christof Bertram and Marc Aubreville and Taryn Donovan and Alexander Bartel and Frauke Wilm and Christian Marzahl and Charles‐Antoine Assenmacher and Kathrin Becker and Mark K. Bennett and Sarah Corner and Brieuc Cossic and Daniela Denk and Martina Dettwiler and Beatriz Garcia Gonzalez and Corinne Gurtner and Ann-Kathrin Haverkamp and Annabelle Heier and Annika Lehmbecker and Sophie Merz and Erica L. Noland and Stephanie Plog and Anja Schmidt and Franziska Sebastian and Dodd G. Sledge and Rebecca C. Smedley and Marco Tecilla and Tuddow Thaiwong and Andrea Fuchs‐Baumgartinger and Don J. Meuten and Katharina Breininger and Matti Kiupel and Andreas Maier and Robert Klopfleisch},
  title = {{Computer-Assisted Mitotic Count Using a Deep Learning-based Algorithm Improves Inter-Observer Reproducibility and Accuracy in canine cutaneous mast cell tumors}},
  year = {2021},
  doi = {10.1101/2021.06.04.446287},
  howpublished = {bioRxiv},
}
Figure