Deep Convolutional Neural Networks For Imaging Data Based Survival Analysis Of Rectal Cancer

Hongming Li, Pamela Boimel, James Janopaul-Naylor, Haoyu Zhong, Ying Xiao, Edgar Ben-Josef, Yong Fan

TL;DR

Researchers have developed a deep learning framework that integrates Convolutional Neural Networks (CNNs) directly with survival analysis to predict rectal cancer recurrence. By using a specialized loss function based on the Cox proportional hazards model and a Spatial Pyramid Pooling (SPP) layer to handle varying tumor sizes, the model outperforms traditional "hand-crafted" radiomic methods in predicting patient outcomes from PET/CT scans.

What Problem Does This Paper Address?

Predicting how a patient with locally advanced rectal cancer will respond to treatment is notoriously difficult. While some patients achieve a complete response after chemoradiation, others face a high risk of local recurrence or metastasis. Current clinical markers (like tumor stage or hemoglobin levels) lack the precision needed to tailor intensive therapies to high-risk individuals.

"Radiomics"—the extraction of data-minable features from medical images—offers a solution, but it faces two major hurdles:

  1. Feature Engineering: Traditional methods rely on "hand-crafted" features (like texture or shape) that may not capture the full complexity of the tumor.
  2. Survival Data Complexity: Most deep learning models are designed for simple classification (e.g., "cancer" vs. "no cancer"). However, clinical studies often involve "censored" data, where we only know a patient was cancer-free up until their last check-up, not necessarily when (or if) the cancer returned. This requires specialized survival analysis models.

Methods

The authors propose a deep learning architecture tailored for survival regression:

  • CNN for Feature Extraction: Instead of using pre-defined features, the model uses three 3D convolutional layers to learn discriminative patterns directly from PET and CT imaging data.
  • Spatial Pyramid Pooling (SPP): A common issue in medical AI is that tumors vary in size, but standard CNNs require fixed-size inputs. The SPP layer allows the network to accept tumors of any dimensions and produce a fixed-length representation for the final layers.
  • Cox Proportional Hazards (CPH) Loss: This is the "secret sauce." Rather than using a standard classification loss, the network is trained to minimize the negative log partial likelihood of a CPH model. This forces the CNN to learn features that specifically correlate with the risk of recurrence over time.
  • Multimodal Fusion: The architecture allows for the simultaneous input of different imaging modalities (CT and PET), which are fused within the network to leverage the strengths of both (anatomical detail from CT and metabolic activity from PET).

Datasets

The model was evaluated on two datasets:

  1. Simulated Dataset: 10,000 images derived from the MNIST digit database (digits "0" and "6"), where survival times were synthetically generated based on a non-linear risk function of the image intensity.
  2. Clinical Dataset: 84 rectal cancer patients who underwent pre-treatment FDG-PET/CT scans. Of these, 14 patients experienced tumor recurrence over a median follow-up of 3 years.

Key Insights and Ideas

  • End-to-End Survival Learning: By integrating the CPH model into the neural network's loss function, the researchers moved away from a two-step process (feature extraction then modeling) to a single, optimized pipeline.
  • Handling Variable Geometry: The use of SPP is a significant improvement over simple resampling or cropping, as it preserves the original spatial characteristics of tumors regardless of their size.
  • The Power of Multimodality: The results confirmed that combining PET and CT data provides a more holistic view of the tumor's "risk profile" than either modality alone.

Key Results

The performance was measured using the Concordance-index (c-index), where 1.0 is a perfect prediction and 0.5 is random chance.

  • Simulated Data: The proposed deep learning method achieved a c-index of 0.96, significantly outperforming standard CPH (0.86) and Random Survival Forests (0.85).
  • Clinical Data (PET/CT): The proposed model reached a c-index of 0.64.
  • In comparison, the standard CPH model scored 0.60 and Random Survival Forests scored 0.58.
  • The model performed better when using combined PET/CT data (0.64) than when using CT (0.62) or PET (0.60) individually.

Limitations and Future Work

The primary limitation acknowledged is the small sample size of the clinical dataset (84 patients, only 14 events). Deep learning typically thrives on thousands of samples; the fact that it outperformed traditional methods on such a small cohort is promising but suggests that results need validation on larger, multi-center datasets.

Future work includes:

  • Fine-tuning hyperparameters to further boost performance.
  • Incorporating non-imaging data, such as genomic markers or clinical blood work, into the fully connected layers of the network to create a truly "pan-diagnostic" survival model.

Summary updated

Abstract

Recent radiomic studies have witnessed promising performance of deep learning techniques in learning radiomic features and fusing multimodal imaging data. Most existing deep learning based radiomic studies build predictive models in a setting of pattern classification, not appropriate for survival analysis studies where some data samples have incomplete observations. To improve existing survival analysis techniques whose performance is hinged on imaging features, we propose a deep learning method to build survival regression models by optimizing imaging features with deep convolutional neural networks (CNNs) in a proportional hazards model. To make the CNNs applicable to tumors with varied sizes, a spatial pyramid pooling strategy is adopted. Our method has been validated based on a simulated imaging dataset and a FDG-PET/CT dataset of rectal cancer patients treated for locally advanced rectal cancer. Compared with survival prediction models built upon hand-crafted radiomic features using Cox proportional hazards model and random survival forests, our method achieved competitive prediction performance.
BibTeX
@inproceedings{li2019deep,
  author = {Hongming Li and Pamela Boimel and James Janopaul-Naylor and Haoyu Zhong and Ying Xiao and Edgar Ben-Josef and Yong Fan},
  title = {{Deep Convolutional Neural Networks For Imaging Data Based Survival Analysis Of Rectal Cancer}},
  booktitle = {ISBI},
  year = {2019},
  url = {https://openreview.net/forum?id=qY9sMez1w9},
}

Loading related content...

Figure