TL;DR
This paper introduces a deep learning-assisted system to improve the accuracy and consistency of counting mitotic figures (MFs) in canine mast cell tumors. By providing pathologists with algorithmic preselection of regions of interest and highlighting potential MFs, the system significantly boosts inter-observer agreement and individual MF detection performance.
What Problem Does This Paper Address?
Accurately counting mitotic figures (MFs) in histological tumor samples is crucial for predicting patient prognosis, especially in canine cutaneous mast cell tumors (ccMCTs). However, this process is notoriously prone to significant variability between pathologists (inter-observer discrepancies) and even by the same pathologist at different times (intra-observer discrepancies). This inconsistency stems from challenges in identifying the correct region within the tumor to count (the "mitotic hotspot") and in accurately distinguishing true MFs from similar-looking cellular structures (imposters). These issues hinder reliable prognostication and standardization of diagnostic practices.
Methods
The study employed a multi-stage approach to evaluate the impact of computer assistance on MF counting.
- Stage 1 (Unaided): Pathologists manually identified regions of interest (MC-ROIs) and counted MFs using their standard methods on whole slide images (WSIs).
- Stage 2 (Partial Computer Assistance): A deep learning algorithm preselected the MC-ROI, highlighting areas with the highest predicted mitotic activity. Pathologists then counted MFs within this preselected region.
- Stage 3 (Full Computer Assistance): In addition to the preselected MC-ROI, the algorithm also visualized individual MF candidates and their confidence scores as an overlay on the WSI. Pathologists reviewed these suggestions and made their final counts.
A key methodological innovation was the development of a "ground truth" dataset using immunohistochemistry for phosphohistone H3 (pHH3). This technique specifically labels cells in the mitotic phase, providing a highly accurate reference for MF identification, which was then compared against both pathologist and algorithmic counts.
Datasets
- Training Data: The deep learning algorithm was trained on a dataset of 44,880 MF annotations from 32 ccMCT WSIs.
- Evaluation Data: The study evaluated 50 ccMCT cases. A subset of 40 cases was used to generate the pHH3-assisted ground truth dataset.
Key Insights and Ideas
- AI as a "Second Pair of Eyes": The study strongly suggests that AI isn't meant to replace pathologists but to augment their capabilities, particularly in tedious and subjective tasks like MF counting.
- Hotspot Identification is Key: The difficulty in finding the "mitotic hotspot" is a major source of variability. Algorithmic preselection of these hotspots proved to be a highly effective intervention.
- Confidence Scores Matter: While not universally embraced, the display of algorithmic confidence scores for MF candidates did influence pathologist decisions, especially when they were uncertain.
- Ground Truth is Crucial for Validation: The use of pHH3-assisted immunohistochemistry provided a robust, objective benchmark that was essential for accurately assessing both algorithmic and human performance.
Key Results
- Improved Inter-Observer Agreement: Computer assistance significantly increased the consistency of MF counts between pathologists. The inter-observer correlation coefficient (ICC) rose from 0.70 (unaided) to 0.92 (full computer assistance).
- Enhanced Individual MF Detection: When using full computer assistance, pathologists' performance in detecting individual MFs improved, with an F1-score increasing from 0.68 to 0.79. This was accompanied by a 38% reduction in false negatives.
- Higher Mitotic Counts with Assistance: Computer-assisted methods consistently yielded higher mitotic counts compared to unaided manual counts, likely due to improved detection of MFs and hotspots.
- Accurate Prognostic Classification: Computer assistance improved the accuracy of classifying cases into prognostic categories based on a standard cut-off value (MC ≥ 5), with accuracy increasing from 75.8% (stage 1) to 91.7% (stage 3) compared to the ground truth.
- Algorithmic Preselection Superiority: Algorithmic MC-ROI preselection was found to be superior to manual selection in identifying mitotic hotspots, with 79% of algorithmically selected ROIs showing higher MCs than manually selected ones.
Limitations and Future Work
- Case Selection Bias: The study's case selection strategy, focusing on cases with high mitotic density, may not fully represent routine diagnostic scenarios. Future studies should verify results on a broader range of cases, especially those with very low MCs.
- Algorithmic Errors: Despite high accuracy, the algorithm did produce some incorrect MF predictions and inappropriate MC-ROI preselections, highlighting the need for expert review.
- Domain Shift: The algorithm's performance on images from different scanners or with varying staining characteristics (domain shift) was not evaluated and requires further validation.
- Prognostic Value: The study focused on MC accuracy and reproducibility, not directly on prognostic outcomes. Future research should investigate how computer-assisted MCs impact patient prognosis.
- MC-ROI Shape: While the rectangular shape was generally accepted, its suitability for all tumor types with variable cellular density or non-neoplastic tissue warrants further investigation.
- Expert Review Level: Determining the optimal degree of expert review required for computer-assisted MCs remains an open question.
- Bias from Confidence Scores: The potential for algorithmic confidence scores to introduce bias (positive or negative) in pathologist decision-making needs further study.
Summary updated