Independent real‐world application of a clinical‐grade automated prostate cancer detection system

Leonard Da Silva, Emílio Marcelo Pereira, Paulo G. O. Salles, Ran Godrich, Rodrigo Ceballos, Jeremy D. Kunz, Adam Casson, Julian Viret, Sarat Chandarlapaty, Carlos Gil Ferreira, Bruno Lemos Ferrari, Brandon Rothrock, Patricia Raciti, Victor E. Reuter, Belma Dogdas, George R. DeMuth, Jillian Sue, Christopher Kanan, Leo Grady, Thomas J. Fuchs, Jorge S. Reis‐FilhoThe Journal of Pathology 2021 · DOI

TL;DR

This study validates Paige Prostate, an AI system for detecting prostate cancer in histopathology slides, on independent, real-world data. The AI demonstrated high sensitivity and negative predictive value, suggesting it can safely flag slides for review, potentially reducing pathologist workload by over 65% and even identifying cancers missed by experienced human pathologists.

What Problem Does This Paper Address?

The diagnosis of prostate cancer from needle biopsies faces significant challenges: variability in pathologist interpretation, high caseloads, and a shortage of qualified pathologists. This can lead to missed diagnoses (false negatives), particularly for small or subtle cancerous foci. The authors aim to evaluate an AI system, Paige Prostate, designed to assist in this diagnostic process by identifying suspicious slides, thereby improving accuracy and efficiency. They specifically test its performance on data from a different country and laboratory than its original development, assessing its real-world applicability.

Methods

The study employed Paige Prostate, a convolutional neural network (CNN) based on a multiple instance learning algorithm, to analyze whole-slide images (WSIs) of transrectal ultrasound-guided (TRUS) prostate needle core biopsies. The AI system was trained to classify slides into two categories: "benign" (requiring no further review) or "suspicious" (requiring additional analysis). The performance of Paige Prostate was compared against the diagnoses of a local pathologist and two independent central pathologists. A "ground truth" diagnosis was established through consensus among the pathologists, supplemented by immunohistochemical (IHC) analysis where disagreements arose. The study evaluated performance at both the "part-specimen" (a biopsy region) and "patient" levels.

Datasets

The study utilized a dataset of 600 transrectal ultrasound-guided (TRUS) prostate needle core biopsy regions ("part-specimens") from 100 consecutive patients. These samples were sourced from the pathology archives of the Instituto Mario Penna in Brazil, representing independent, real-world data not involved in the original development or validation of the Paige Prostate system.

Key Insights and Ideas

  • Real-world Generalizability: The AI system, developed and validated elsewhere, performed well on independent data from a different country and laboratory, suggesting its potential for broad clinical adoption.
  • Safety Net for Missed Diagnoses: The AI's high sensitivity and NPV mean it's unlikely to miss actual cancer cases, making it a reliable tool for pre-screening.
  • Efficiency Gains: The potential for a significant reduction in diagnostic time (estimated at 65.5%) is a major practical benefit, addressing the pressure on pathology departments.
  • Augmenting Pathologist Expertise: The AI not only improved efficiency but also identified cases of prostate cancer that were initially missed by experienced pathologists, highlighting its value as an assistive tool.
  • Defining "Suspicious" Appropriately: The AI flagged a reasonable number of slides as suspicious, and a significant portion of these were benign but required further investigation, indicating a good balance between sensitivity and specificity.

Key Results

  • Part-Specimen Level Performance: Paige Prostate achieved high sensitivity (0.99) and NPV (1.0), with a specificity of 0.93.
  • Patient Level Performance: At the patient level, sensitivity and NPV were optimal (1.0), with a specificity of 0.78.
  • Missed Diagnoses Identified: The AI identified four additional patients whose diagnoses were upgraded from benign/suspicious to malignant. It also flagged 11 new suspicious part-specimens that were initially diagnosed as benign by three experienced pathologists.
  • Time Savings: The AI-based test provided an estimated 65.5% reduction in diagnostic time for the analyzed material.
  • False Positives: Of the slides flagged as suspicious by the AI, 27 were ultimately diagnosed as benign. These included cases of atrophy, adenosis, and post-atrophic hyperplasia, indicating the AI's tendency to err on the side of caution.

Limitations and Future Work

  • Ground Truth Definition: The "ground truth" diagnosis relied on consensus and IHC, which, while robust, is not infallible.
  • AI System Optimization: The study used a specific version of Paige Prostate, and further refinements might improve specificity and PPV.
  • Incomplete Data: Not all part-specimens were analyzed for a subset of patients due to technical issues, potentially affecting patient-level performance estimates.
  • Binary Classification: The current AI version provides a binary "benign" or "suspicious" classification and does not offer descriptive diagnoses (e.g., atrophy) or grading. Future work should focus on developing AI systems capable of these more nuanced outputs.
  • Time Reduction Inference: The reported time reduction is an estimate based on average review times and may not fully account for the time involved in slide scanning and AI processing.
  • Future Directions: The authors suggest future studies should explore the feasibility and optimal implementation of AI systems like Paige Prostate in both academic and private laboratory settings, and investigate their use in automated quality control and screening. They also highlight the need for regulatory frameworks to support AI deployment in pathology.

Summary updated

Abstract

Artificial intelligence (AI)-based systems applied to histopathology whole-slide images have the potential to improve patient care through mitigation of challenges posed by diagnostic variability, histopathology caseload, and shortage of pathologists. We sought to define the performance of an AI-based automated prostate cancer detection system, Paige Prostate, when applied to independent real-world data. The algorithm was employed to classify slides into two categories: benign (no further review needed) or suspicious (additional histologic and/or immunohistochemical analysis required). We assessed the sensitivity, specificity, positive predictive values (PPVs), and negative predictive values (NPVs) of a local pathologist, two central pathologists, and Paige Prostate in the diagnosis of 600 transrectal ultrasound-guided prostate needle core biopsy regions ('part-specimens') from 100 consecutive patients, and to ascertain the impact of Paige Prostate on diagnostic accuracy and efficiency. Paige Prostate displayed high sensitivity (0.99; CI 0.96-1.0), NPV (1.0; CI 0.98-1.0), and specificity (0.93; CI 0.90-0.96) at the part-specimen level. At the patient level, Paige Prostate displayed optimal sensitivity (1.0; CI 0.93-1.0) and NPV (1.0; CI 0.91-1.0) at a specificity of 0.78 (CI 0.64-0.89). The 27 part-specimens considered by Paige Prostate as suspicious, whose final diagnosis was benign, were found to comprise atrophy (n = 14), atrophy and apical prostate tissue (n = 1), apical/benign prostate tissue (n = 9), adenosis (n = 2), and post-atrophic hyperplasia (n = 1). Paige Prostate resulted in the identification of four additional patients whose diagnoses were upgraded from benign/suspicious to malignant. Additionally, this AI-based test provided an estimated 65.5% reduction of the diagnostic time for the material analyzed. Given its optimal sensitivity and NPV, Paige Prostate has the potential to be employed for the automated identification of patients whose histologic slides could forgo full histopathologic review. In addition to providing incremental improvements in diagnostic accuracy and efficiency, this AI-based system identified patients whose prostate cancers were not initially diagnosed by three experienced histopathologists. © 2021 The Authors. The Journal of Pathology published by John Wiley & Sons, Ltd. on behalf of The Pathological Society of Great Britain and Ireland.
BibTeX
@article{silva2021independent,
  author = {Leonard Da Silva and Emílio Marcelo Pereira and Paulo G. O. Salles and Ran Godrich and Rodrigo Ceballos and Jeremy D. Kunz and Adam Casson and Julian Viret and Sarat Chandarlapaty and Carlos Gil Ferreira and Bruno Lemos Ferrari and Brandon Rothrock and Patricia Raciti and Victor E. Reuter and Belma Dogdas and George R. DeMuth and Jillian Sue and Christopher Kanan and Leo Grady and Thomas J. Fuchs and Jorge S. Reis‐Filho},
  title = {{Independent real‐world application of a clinical‐grade automated prostate cancer detection system}},
  journal = {The Journal of Pathology},
  year = {2021},
  doi = {10.1002/path.5662},
}
Figure