Abstract
Recent radiomic studies have witnessed promising performance of deep learning techniques in learning radiomic features and fusing multimodal imaging data. Most existing deep learning based radiomic studies build predictive models in a setting of pattern classification, not appropriate for survival analysis studies where some data samples have incomplete observations. To improve existing survival analysis techniques whose performance is hinged on imaging features, we propose a deep learning method to build survival regression models by optimizing imaging features with deep convolutional neural networks (CNNs) in a proportional hazards model. To make the CNNs applicable to tumors with varied sizes, a spatial pyramid pooling strategy is adopted. Our method has been validated based on a simulated imaging dataset and a FDG-PET/CT dataset of rectal cancer patients treated for locally advanced rectal cancer. Compared with survival prediction models built upon hand-crafted radiomic features using Cox proportional hazards model and random survival forests, our method achieved competitive prediction performance.
Introduction
Predicting treatment response and recurrence risk in locally advanced rectal cancer remains a significant clinical challenge. While some patients respond well to chemoradiation, others face high risks of local recurrence or metastasis. Traditional clinical markers often lack the precision required for personalized therapy. Radiomics—the high-throughput extraction of quantitative features from medical imaging—offers a potential solution. However, traditional radiomics relies on "hand-crafted" features that may fail to capture complex tumor characteristics. Furthermore, standard deep learning models are typically optimized for classification rather than survival analysis, which must account for "censored" data (incomplete observations).
Proposed Methodology
The proposed framework integrates deep learning with survival analysis in an end-to-end pipeline. The architecture utilizes 3D Convolutional Neural Networks (CNNs) to automatically learn discriminative features from multimodal PET and CT imaging data.
Key technical innovations include:
- Spatial Pyramid Pooling (SPP): To address the variability in tumor sizes across patients, an SPP layer is incorporated. This allows the network to accept input volumes of varying dimensions while producing a fixed-length feature vector for the subsequent fully connected layers, preserving spatial characteristics without the need for distortive resampling.
- Cox Proportional Hazards (CPH) Loss: Instead of standard classification losses, the network is trained by minimizing the negative log partial likelihood of a CPH model. This ensures that the learned features are directly optimized to predict the risk of recurrence over time.
- Multimodal Fusion: The model simultaneously processes CT (anatomical detail) and PET (metabolic activity) data, fusing these streams to create a comprehensive risk profile.
Experimental Evaluation
The model was validated on two distinct datasets:
- Simulated Dataset: 10,000 images derived from MNIST (digits "0" and "6"), with survival times generated via a non-linear risk function.
- Clinical Dataset: Pre-treatment FDG-PET/CT scans from 84 rectal cancer patients, with 14 recorded recurrence events over a median 3-year follow-up.
Key Results
Performance was evaluated using the Concordance-index (c-index), where 1.0 represents perfect prediction. On the simulated dataset, the proposed method achieved a c-index of 0.96, significantly outperforming standard CPH (0.86) and Random Survival Forests (0.85).
In the clinical PET/CT dataset, the deep learning approach consistently outperformed traditional methods.
The results demonstrate that the proposed model achieves its highest performance (c-index 0.64) when utilizing combined PET/CT data, highlighting the value of multimodal fusion over single-modality analysis.
Conclusion and Future Work
This study demonstrates the feasibility of integrating CNNs with survival analysis for cancer prognosis. By moving beyond hand-crafted features and fixed-size inputs, the model provides a more robust framework for radiomic analysis. While the small clinical sample size is a limitation, the superior performance over traditional models is encouraging. Future research will focus on validating these results in larger multi-center cohorts and incorporating non-imaging data, such as genomic and clinical markers, to further enhance predictive accuracy.
Summary updated