Interpretable survival prediction for colorectal cancer using deep learning

Yun Liu, Ellery Wulczyn, David F. Steiner, Melissa Moran, Markus Plass, Robert Reihs, Fraser Tan, Isabelle Flament-Auvigne, Trissia Brown, Peter Regitnig, Po-Hsuan Cameron Chen, Narayan Hegde, Apaar Sadhwani, Robert MacDonald, Benny Ayalew, Gregory S. Corrado, Lily H. Peng, Daniel Tse, Heimo Müller, Zhaoyang Xu, Yun Liu 0013, Martin C. Stumpe, Kurt Zatloukal, Craig H. Mermel

TL;DR

This paper introduces a deep learning system (DLS) for predicting survival in colorectal cancer patients directly from histopathology images. Crucially, it develops a method to generate human-interpretable "clustering-derived" features that explain the DLS's predictions and are themselves prognostic, offering a path towards more trustworthy AI in medicine.

What Problem Does This Paper Address?

Predicting patient survival is vital for guiding cancer treatment decisions. While traditional methods like TNM staging provide a baseline, they often fail to capture the full spectrum of patient outcomes due to inherent variability. Machine learning, particularly deep learning, has shown promise in extracting complex patterns from histopathology images for prognostic prediction. However, a major hurdle has been the "black box" nature of these models, making it difficult to understand why a prediction is made and to trust the results. This lack of interpretability hinders clinical adoption, as doctors need to understand the basis of a prediction to integrate it into their decision-making. This paper aims to bridge this gap by developing an interpretable deep learning system for colorectal cancer survival prediction.

Methods

The core of this work is a two-stage deep learning system:

  1. Tumor Segmentation Model: A convolutional neural network (CNN) was trained to identify tumor regions within whole-slide images. This model achieved a high AUC of 0.985, ensuring that subsequent analysis focused on relevant areas.
  2. Prognostic Model: This CNN was trained to predict disease-specific survival (DSS) using image patches sampled from the tumor regions identified by the segmentation model. The model uses a Cox regression layer to output a case-level risk score.

The key innovation lies in interpreting the DLS. Instead of relying solely on traditional clinicopathologic features (which explained only 18% of the DLS's prediction variance), the authors developed a method to generate human-interpretable "clustering-derived" features. This involved:

  • Using a deep learning-based image-similarity model to generate embeddings from image patches.
  • Clustering these embeddings to create groups of visually similar image patches.
  • Quantifying the presence of these clusters within each patient's tumor.

These clustering-derived features were then used to explain the DLS predictions and were also evaluated for their independent prognostic value.

Datasets

The study utilized archived, de-identified histopathology slides and associated clinical data from the Medical University of Graz, Austria.

  • Development Set: Cases from 1984-2007 were split into training, tuning, and validation set 1.
  • Training: 2439 cases
    • Tune: 1213 cases
    • Validation Set 1: 1239 cases (with 10-35 years of follow-up)
  • Temporal Validation Set: Cases from 2008-2013 served as validation set 2.
  • Validation Set 2: 738 cases (with 5-9 years of follow-up)

This setup allowed for evaluation of both performance and temporal generalization.

Key Insights and Ideas

  • Deep learning can predict survival from histology alone: The DLS achieved significant risk stratification, demonstrating that complex patterns in histopathology images hold substantial prognostic information.
  • Traditional features are insufficient: Clinicopathologic features explained only a small fraction (18%) of the DLS's predictive variance, highlighting the need for more sophisticated analysis.
  • "Clustering-derived" features unlock interpretability: By clustering image embeddings, the authors created novel, human-interpretable features that explained a majority (73-80%) of the DLS's prediction variance. This is a significant step towards understanding what the deep learning model is "seeing."
  • A novel prognostic feature identified: One particular clustering-derived feature, termed "Tumor-Adipose Feature" (TAF), characterized by tumor cells adjacent to adipose tissue, was strongly associated with high DLS risk scores and was independently prognostic.
  • Reproducible identification of novel features: Pathologists and even non-pathologists could identify TAF with high accuracy, suggesting that these machine-learned features can be translated into a form that clinicians can use.

Key Results

  • The DLS achieved a 5-year disease-specific survival AUC of 0.70 (95% CI 0.66-0.73) and 0.69 (95% CI 0.64-0.72) on the two validation sets, respectively.
  • The clinicopathologic features explained only 18% of the variance in DLS scores.
  • The clustering-derived features explained the majority of the variance in DLS scores, with 73% to 80% R² in the validation sets.
  • A subset of 10 clustering-derived features explained 57% to 61% of the variance.
  • The Tumor-Adipose Feature (TAF) was independently prognostic and could be identified by pathologists with 87.0-95.5% accuracy.

Limitations and Future Work

  • Retrospective study design: Treatment pathways are a confounding factor that is difficult to control for. Progression-free survival, a less confounded endpoint, was not available at the required scale.
  • Temporal validation: While temporal validation was performed, validation in geographically diverse cohorts would further strengthen generalization.
  • Limited evaluation of known prognostic factors: The study could not evaluate the association of the DLS with several known prognostic factors (e.g., tumor budding, molecular profiles).
  • Future work:
    • Prospective studies are needed to evaluate the impact of DLS-informed treatment decisions.
  • Further research is warranted to understand the biological significance of TAF and other clustering-derived features.
  • Developing guidelines for consistent scoring of TAF by pathologists is necessary.
  • Exploring embeddings from different machine learning models could identify additional features.
  • Investigating the approximately 20% of unexplained variance in DLS scores.

Summary updated

Abstract

Deriving interpretable prognostic features from deep-learning-based prognostic histopathology models remains a challenge. In this study, we developed a deep learning system (DLS) for predicting disease specific survival for stage II and III colorectal cancer using 3,652 cases (27,300 slides). When evaluated on two validation datasets containing 1,239 cases (9,340 slides) and 738 cases (7,140 slides) respectively, the DLS achieved a 5-year disease-specific survival AUC of 0.70 (95%CI 0.66-0.73) and 0.69 (95%CI 0.64-0.72), and added significant predictive value to a set of 9 clinicopathologic features. To interpret the DLS, we explored the ability of different human-interpretable features to explain the variance in DLS scores. We observed that clinicopathologic features such as T-category, N-category, and grade explained a small fraction of the variance in DLS scores (R2=18% in both validation sets). Next, we generated human-interpretable histologic features by clustering embeddings from a deep-learning based image-similarity model and showed that they explain the majority of the variance (R2 of 73% to 80%). Furthermore, the clustering-derived feature most strongly associated with high DLS scores was also highly prognostic in isolation. With a distinct visual appearance (poorly differentiated tumor cell clusters adjacent to adipose tissue), this feature was identified by annotators with 87.0-95.5% accuracy. Our approach can be used to explain predictions from a prognostic deep learning model and uncover potentially-novel prognostic features that can be reliably identified by people for future validation studies.
BibTeX
@article{liu2021interpretable,
  author = {Yun Liu and Ellery Wulczyn and David F. Steiner and Melissa Moran and Markus Plass and Robert Reihs and Fraser Tan and Isabelle Flament-Auvigne and Trissia Brown and Peter Regitnig and Po-Hsuan Cameron Chen and Narayan Hegde and Apaar Sadhwani and Robert MacDonald and Benny Ayalew and Gregory S. Corrado and Lily H. Peng and Daniel Tse and Heimo Müller and Zhaoyang Xu and Yun Liu and Martin C. Stumpe and Kurt Zatloukal and Craig H. Mermel},
  title = {{Interpretable survival prediction for colorectal cancer using deep learning}},
  journal = {Npj Digit. Medicine},
  year = {2021},
  doi = {10.1038/s41746-021-00427-2},
}
Figure