Evaluating a deep learning AI algorithm for detecting residual prostate cancer on MRI after focal therapy

David G. Gelikman, Stephanie A. Harmon, Alexander P. Kenigsberg, Yan Mee Law, Enis C. Yılmaz, Maria J. Merino, Bradford J. Wood, Peter L. Choyke, Peter Pinto, Barış TürkbeyBJUI 2024 · DOI

TL;DR

This paper evaluates a deep learning AI algorithm, originally trained on untreated prostate MRIs, for detecting residual prostate cancer after focal therapy (FT). Despite limitations, the AI showed moderate sensitivity, suggesting potential for post-treatment cancer detection, especially with further training on post-FT specific data.

What Problem Does This Paper Address?

Detecting residual prostate cancer (PCa) after focal therapy (FT) is challenging. FT aims to destroy localized PCa, but it often distorts the prostate's anatomy, making it difficult for radiologists to accurately interpret follow-up MRI scans. While AI has shown promise in detecting PCa in untreated prostates, its effectiveness in the post-FT setting, where anatomical changes are significant, is largely unexplored. This study aims to bridge that gap by assessing a pre-existing AI model's ability to identify residual PCa on biparametric MRI (bpMRI) after FT.

Methods

The researchers utilized a retrospective cohort of 40 patients who underwent post-FT prostate bpMRIs from an IRB-approved clinical trial. They employed a previously developed 3D U-Net-based deep neural network AI model. This AI was originally trained on a diverse dataset of MRIs from treatment-naïve patients and is designed to detect suspicious lesions using T2-weighted images, apparent diffusion coefficient (ADC) maps, and high b-value diffusion-weighted images. The AI's output was a prediction map highlighting suspicious lesions. These predictions were then compared against MRI/transrectal ultrasound (TRUS) fusion-guided and systematic prostate biopsies to determine performance metrics like sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy. A patient-level analysis was conducted, defining true positives, false positives, false negatives, and true negatives based on biopsy results and AI predictions.

Datasets

The study evaluated a dataset of 40 post-FT prostate bpMRIs from an IRB-approved clinical trial (NCT03354416). The AI model itself was originally trained on a diverse MRI dataset obtained from treatment-naïve patients. No new datasets were introduced or benchmarked in this paper.

Key Insights and Ideas

  • AI's Potential Beyond "Normal" Anatomy: The study highlights that even an AI model trained on "ideal" (treatment-naïve) prostates can offer valuable insights in a challenging post-treatment scenario. This suggests a foundational capability that can be built upon.
  • The Challenge of Anatomical Distortion: The significant anatomical distortion caused by focal therapy is a major hurdle for both human interpretation and AI models, underscoring the need for specialized training data.
  • bpMRI vs. DCE-MRI: The AI's reliance on bpMRI (T2-weighted, ADC, DWI) is a limitation, as post-FT MRI analysis often incorporates dynamic contrast-enhanced (DCE) imaging. This implies that future AI development for this specific application might need to integrate DCE data, potentially requiring a complete model overhaul.
  • Biopsy as a "Noisy" Ground Truth: Using targeted and systematic biopsies as the gold standard for ground truth is inherently limited. It doesn't capture the entire prostate and can introduce selection bias, making it difficult to definitively label AI predictions as truly false positives or negatives.

Key Results

The AI model achieved the following performance metrics on the 40 post-FT patients:

  • Sensitivity: 47.4% (9/19)
  • Specificity: 28.6% (6/21)
  • Positive Predictive Value (PPV): 37.5% (9/24)
  • Negative Predictive Value (NPV): 37.5% (6/16)
  • Overall Accuracy: 37.5% (15/40)

Out of 40 patients, 25 had PCa at biopsy. The AI made 33 unique lesion predictions across 24 patients. The breakdown of predictions was:

  • True Positives (TP): 9 patients (22.5%)
  • False Positives (FP): 15 patients (37.5%)
  • False Negatives (FN): 10 patients (25%)
  • True Negatives (TN): 6 patients (15%)

Limitations and Future Work

The authors acknowledge several limitations:

  • Training Data: The AI was trained on treatment-naïve patients, not post-FT images, which likely impacted its performance.
  • bpMRI Only: The AI does not utilize dynamic contrast-enhanced (DCE) MRI, a common sequence for post-FT evaluation. Future work should explore incorporating DCE data.
  • Ground Truth: The reliance on targeted and systematic biopsies as the ground truth is a limitation. Whole-gland pathology would provide a more definitive assessment but is not always available.
  • Lack of Standardized Interpretation: A standard method for radiologist interpretation of post-FT images is still developing, making direct comparison challenging.

Future work should focus on training AI models with datasets specifically from post-FT patients, potentially incorporating DCE imaging. Comparing future AI algorithms to standardized interpretation systems for post-FT images is also recommended. The study serves as a foundational step, motivating further research to improve general AI models for PCa detection in this challenging post-treatment context.

Summary updated

Abstract

Advancements in artificial intelligence (AI) have shown promise in standardizing medical imaging evaluations, particularly in detecting prostate cancer (PCa) on MRI.1 Though MRI-based AI algorithms have been developed to detect PCa in untreated glands,2, 3 little research exists on the efficacy of such models after prostate ablation. While focal therapy (FT) targets and destroys localized PCa, it usually distorts prostate anatomy, making it difficult to evaluate on MRI.4 Our study investigates the efficacy of a biparametric MRI (bpMRI)-based deep learning algorithm for post-FT PCa identification. This retrospective cohort study utilized post-FT prostate bpMRIs from an IRB-approved clinical trial (NCT03354416). MRIs were evaluated with a previously developed AI model, a 3D U-Net-based deep neural network that can detect suspicious lesions on untreated prostate bpMRIs based on T2-weighted images, apparent diffusion coefficient maps and high b-value diffusion-weighted images (Figure 1A–C).5 This algorithm was originally trained using a diverse MRI dataset obtained from treatment-naïve patients. AI output consisted of PCa-suspicious lesion prediction maps overlayed on T2-weighted MRI (Figure 1D). Predictions were compared to MRI/transrectal ultrasound fusion-guided and systematic prostate biopsies. A patient-level analysis was performed where if at least one location containing Gleason Grade ≥1 disease was detected by the AI, this was a true positive. If an AI prediction was made in an area that turned out to be benign on biopsy, this was a false positive, even if biopsy revealed malignancy in a different region of the prostate. Patients with biopsy-proven PCa lesions that were not predicted by AI were false negatives. If AI made no predictions in a patient with a fully benign prostate biopsy, this was a true negative. AI performance metrics included sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV) and overall accuracy. Of the 40 included patients, the median time to post-FT MRI was 2.5 years, and 25 patients had PCa at biopsy. AI made 33 unique lesion predictions across 24 patients. Of these, 16 patients (67%) had one lesion prediction, 7 patients (29%) had two lesion predictions and 1 patient (4%) had three lesion predictions. Across all AI predictions in this cohort, 9 patients (22.5%) had true positives, 15 patients (37.5%) had false positives, 10 patients (25%) had false negatives and 6 patients (15%) had true negatives. The AI's overall sensitivity was 47.4% with a specificity of 28.6%. The PPV and NPV were both 37.5%. Overall, the AI achieved an accuracy of 37.5%. The performance characteristics of this model are listed in Table 1. Our AI reached a moderate level of sensitivity. Despite low specificity and overall accuracy, this is a noteworthy finding, as this AI algorithm was trained on treatment-naïve glands and not on post-FT images. The 47% sensitivity rate underscores its potential and future effectiveness if specific training with post-FT images could be achieved. This compares favourably to radiologist interpretations of MRI post-FT, with some series demonstrating sub-50% sensitivity.6, 7 Additionally, post-FT MRI analysis typically relies on dynamic contrast-enhanced (DCE) imaging over typical bpMRI sequences.8 However, our AI is based on bpMRI and does not include DCE MRI, so additional training on DCE data would likely require a complete renovation of the AI model. Besides reliance on bpMRI, another limitation was the use of targeted and systematic prostate biopsies as the ground truth. While having whole gland specimens could have demonstrated whether lesions detected only by the AI were true or false positives, this may have resulted in a selection bias in our study population, as not all patients undergo surgery. Additionally, targeted biopsies were performed based on original prospective MRI read-outs and not AI predictions. A standard-of-care system for radiologist analysis of post-FT images has yet to be established, although the PI-FAB system shows promise.8 Future AI algorithms will merit comparison to such standardized systems of interpretation. In conclusion, the performance of this model in the post-FT setting is noteworthy given the limitations of its training data and may already perform similarly to radiologist reads, although further research is necessary. This study provides motivation to improve the performance of a general AI model for prostate cancer lesion detection and serves as an initial step in understanding the potential role of AI in PCa detection in post-FT patients. The authors declare no conflict of interest.
BibTeX
@inproceedings{gelikman2024evaluating,
  author = {David G. Gelikman and Stephanie A. Harmon and Alexander P. Kenigsberg and Yan Mee Law and Enis C. Yılmaz and Maria J. Merino and Bradford J. Wood and Peter L. Choyke and Peter Pinto and Barış Türkbey},
  title = {{Evaluating a deep learning AI algorithm for detecting residual prostate cancer on MRI after focal therapy}},
  booktitle = {BJUI},
  year = {2024},
  doi = {10.1002/bco2.373},
}
Figure