Exploiting segmentation labels and representation learning to forecast therapy response of PDAC patients

Alexander Ziller, Ayhan Can Erdur, Friederike Jungmann, Daniel Rueckert, Rickmer Braren, Georgios Kaissis

Abstract

The prediction of pancreatic ductal adenocarcinoma (PDAC) therapy response is a clinically challenging and important task in this high-mortality tumour entity. The training of neural networks able to tackle this challenge is impeded by a lack of large datasets and the difficult anatomical localisation of the pancreas. Here, we propose a hybrid deep neural network pipeline to predict tumour response to initial chemotherapy which is based on the Response Evaluation Criteria in Solid Tumors (RECIST) score, a standardised method for cancer response evaluation by clinicians as well as tumour markers, and clinical evaluation of the patients. We leverage a combination of representation transfer from segmentation to classification, as well as localisation and representation learning. Our approach yields a remarkably data-efficient method able to predict treatment response with a ROC-AUC of 63.7% using only 477 datasets in total.

Introduction

Pancreatic ductal adenocarcinoma (PDAC) remains one of the most lethal malignancies, where early prediction of chemotherapy response is critical for personalized treatment. However, developing robust AI models for this task faces significant hurdles. First, medical datasets with high-quality outcome labels are scarce. Second, the pancreas occupies a very small portion of a full abdominal CT scan, creating a "needle in a haystack" problem where models struggle to focus on relevant features. Finally, while advanced architectures like transformers can handle small regions, they typically require massive datasets unavailable in this domain. This work introduces a multi-stage pipeline designed to overcome these challenges through efficient localization and representation learning.

Methodology

The proposed pipeline mimics a radiologist's workflow by progressively narrowing the focus from the whole body to the tumor site.

Multi-Stage Pipeline

The architecture consists of three primary stages:

  1. Stage I: Slice Localization: A 2.5D approach using ResNet50 and LSTM identifies slices along the Z-axis that contain the pancreas, significantly reducing the search space.
  2. Stage II: Segmentation and Cropping: A 3D segmentation network (DynU-Net) generates precise masks and bounding boxes. These masks are used for "Segmentation Forwarding," where they are fed as an additional input channel to the final classifier to provide geometric context.
  3. Stage III: Classification with Representation Learning: The final classifier (ResNet50) utilizes weights transferred from the segmentation task. It is trained using a combination of Binary Cross Entropy and Triplet Loss to better separate responders from non-responders in the feature space.
Overview of the multi-stage pipeline: (A) Slice localization, (B) Segmentation and cropping, and (C) Classification with transfer learning and triplet loss.

Data Strategy

To address data scarcity, the model leverages two datasets:

  • Medical Segmentation Decathlon (MSD): 281 abdominal CT scans used for training the localization and segmentation components.
  • Clinical Dataset: 477 in-house CT scans with RECIST-based response labels (420 for training, 57 for testing).

Results and Discussion

The integration of localization and segmentation-based transfer learning led to substantial performance gains. A baseline 3D ResNet50 achieved a near-random Matthews Correlation Coefficient (MCC) of 5.6%. Implementing slice-informed cropping improved the MCC to 18.8%. The full pipeline, including segmentation forwarding and triplet loss, reached an MCC of 33.1%, with an Accuracy of 67.2% and an AUC-ROC of 63.7%.

Visualization of the cropped pancreatic region with the segmentation mask used as input for the final classification stage.

Key insights include:

  • Transfer Learning Efficiency: Reusing segmentation weights provided a 2.5x training speedup and improved feature extraction.
  • Geometric Guidance: Providing the model with explicit segmentation masks allowed it to focus on tumor-specific textures rather than background noise.
  • Metric Robustness: The use of MCC ensured that the model's performance was evaluated fairly despite class imbalances common in clinical data.

Conclusion

This study demonstrates that a data-efficient, multi-stage approach can successfully predict PDAC treatment response. By "recycling" knowledge from segmentation tasks and focusing the model's attention on localized regions, we can achieve meaningful predictions even with limited clinical datasets. Future work will explore the integration of multi-modal data, such as genetic markers and longitudinal scans, to further enhance predictive accuracy and clinical utility.

Summary updated

Abstract

The prediction of pancreatic ductal adenocarcinoma therapy response is a clinically challenging and important task in this high-mortality tumour entity. The training of neural networks able to tackle this challenge is impeded by a lack of large datasets and the difficult anatomical localisation of the pancreas. Here, we propose a hybrid deep neural network pipeline to predict tumour response to initial chemotherapy which is based on the Response Evaluation Criteria in Solid Tumors (RECIST) score, a standardised method for cancer response evaluation by clinicians as well as tumour markers, and clinical evaluation of the patients. We leverage a combination of representation transfer from segmentation to classification, as well as localisation and representation learning. Our approach yields a remarkably data-efficient method able to predict treatment response with a ROC-AUC of 63.7% using only 477 datasets in total.
BibTeX
@inproceedings{ziller2023exploiting,
  author = {Alexander Ziller and Ayhan Can Erdur and Friederike Jungmann and Daniel Rueckert and Rickmer Braren and Georgios Kaissis},
  title = {{Exploiting segmentation labels and representation learning to forecast therapy response of PDAC patients}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230324},
}
Figure