TL;DR
This paper details the results of the HER2 Scoring Challenge, a competition designed to benchmark automated algorithms for scoring HER2 protein expression in breast cancer tissue images. The contest highlights the significant potential of AI in pathology, with automated methods showing performance comparable to, and in some cases exceeding, human experts, paving the way for more objective and efficient cancer diagnostics.
What Problem Does This Paper Address?
Accurately determining HER2 protein expression in breast cancer tissue is crucial for guiding treatment decisions, particularly regarding the use of Herceptin. Traditionally, this is done through immunohistochemistry (IHC) followed by visual scoring by pathologists. However, this manual process is subjective and prone to inter-observer variability, leading to potential inconsistencies in diagnosis and treatment. The paper addresses the need for more objective, reproducible, and efficient methods by evaluating the performance of automated algorithms designed to perform this scoring task.
Methods
The core of this work is the HER2 Scoring Challenge, which involved a competitive evaluation of automated algorithms. The challenge utilized a dataset of digitized whole slide images (WSIs) of breast carcinoma stained with H&E and IHC for HER2. Participating algorithms were tasked with predicting HER2 scores on an unseen subset of the data. The evaluation was based on three criteria: agreement points (measuring how closely predictions matched the ground truth), weighted confidence (a measure of the algorithm's certainty), and combined points (a holistic score). A "Man vs. Machine" event was also conducted to directly compare the performance of top automated methods against expert pathologists on a subset of cases.
Datasets
The contest utilized a dataset comprising 172 whole slide images (WSIs) from 86 cases of invasive breast carcinoma. These WSIs included both Hematoxylin & Eosin (H&E) and HER2-stained immunohistochemistry (IHC) slides. A portion of these images (52 cases) formed the training dataset, while a separate set of 28 cases was used for off-site testing. The ground truth (GT) for scoring was established by at least two specialist consultant histopathologists.
Key Insights and Ideas
- AI is closing the gap with human experts: The contest demonstrated that automated algorithms are not only capable of scoring HER2 but are achieving performance levels comparable to, and in some cases surpassing, experienced pathologists.
- Objectivity is key: The inherent subjectivity of manual HER2 scoring is a significant limitation. Automated methods offer a path towards greater objectivity and reproducibility, which is critical for consistent patient care.
- The "2+" category is still a challenge: Cases with a HER2 score of 2+ are equivocal and typically require further testing (FISH). Both human and automated methods struggled with this category, highlighting the need for continued research and refinement in this area.
- Deep learning dominates: The majority of top-performing algorithms employed deep learning architectures, particularly Convolutional Neural Networks (CNNs), underscoring their effectiveness in image analysis tasks within pathology.
Key Results
- Competitive performance: Many automated algorithms achieved high accuracy, with some performing comparably to histopathologists.
- Man vs. Machine: In the Man vs. Machine event, one automated method slightly outperformed the top-performing human participant.
- Agreement on extreme scores: There was 100% agreement with the ground truth for cases scoring 3+ among both human and automated methods, meaning no patient would have been denied treatment. Similarly, for 0 and 1+ scores, over-calling never exceeded 2+, ensuring patients received appropriate consideration for treatment.
- Challenges with 2+ scores: For cases scoring 2+, both humans and machines showed disparities with the ground truth. While machines overcalled 2+ as 3+ in only one instance, humans did so on three occasions. Conversely, automated methods erroneously predicted 2+ as 1+ or 0 in some instances, which could deny further assessment.
Limitations and Future Work
- Dataset size and diversity: The contest data was collected from a single site using a single scanner. Future work should involve collecting data from multiple pathology labs with varying staining protocols and scanners to improve algorithm generalizability and overcome overfitting.
- Computational cost: Some methods required significant processing time (over three hours per case), which is not feasible for routine diagnostic settings. Optimization for speed is crucial.
- FISH integration: While the contest focused on IHC scoring, integrating FISH results into automated methods could provide a more complete picture, especially for 2+ cases.
- H&E slide utilization: Most algorithms avoided using H&E slides, which pathologists often use to map tumor regions. Future methods could explore the synergistic use of both H&E and IHC for improved accuracy.
- Refinement for equivocal cases: Further fine-tuning of AI algorithms is needed to accurately handle the challenging 2+ cases.
- Broader AI applications: The success of this contest suggests potential for AI in automating the scoring of other biomarkers like PD-1, PD-L1, ALK, and ROS1.
Summary updated