Exploiting Segmentation Labels and Representation Learning to Forecast Therapy Response of PDAC Patients

Alexander Ziller, Ayhan Can Erdur, Friederike Jungmann, Daniel Rueckert, Rickmer Braren, Georgios Kaissis

TL;DR

Researchers have developed a multi-stage deep learning pipeline that predicts how pancreatic cancer (PDAC) patients will respond to chemotherapy using only baseline CT scans. By cleverly "recycling" knowledge from a separate organ segmentation task and using triplet loss for better feature representation, the team achieved a significant performance boost on a small dataset of fewer than 500 patients.

What Problem Does This Paper Address?

Pancreatic ductal adenocarcinoma (PDAC) is one of the deadliest cancers, and predicting whether a patient will respond to chemotherapy is a "holy grail" for personalizing treatment. However, training AI to do this is notoriously difficult for three reasons:

  1. Data Scarcity: Large, labeled medical datasets for specific treatment outcomes are rare.
  2. The "Needle in a Haystack" Problem: A full abdominal CT scan is massive, but the pancreas (and the tumor) occupies only a tiny fraction of that volume. Standard models often "miss" the relevant pixels.
  3. Architecture Trade-offs: While attention-based models can focus on small areas, they usually require massive amounts of data—which we don't have in this field.

The authors aim to bridge this gap by using a hybrid approach that helps the model "find" the tumor and "understand" its shape before trying to predict its future behavior.

Methods

The authors propose a three-stage pipeline designed to mimic the way a human radiologist approaches a scan:

  • Stage I: Slice Classification (The Z-Axis): To avoid processing the entire 3D volume at once, they use a 2.5D approach (ResNet50 + LSTM) to identify which horizontal slices actually contain the pancreas. This narrows the search space along the patient's height.
  • Stage II: Segmentation & Cropping (The X/Y Axes): They use a 3D segmentation network (DynU-Net) to create a bounding box around the organ. Crucially, they use "Segmentation Forwarding"—feeding the predicted segmentation masks back into the final classifier as an extra channel to provide "assistive" geometric information.
  • Stage III: Classification with Representation Learning: The final ResNet50 classifier doesn't start from scratch. It reuses the weights from the segmentation model (Transfer Learning). To further refine the model, they use a Triplet Loss function, which forces the model to group "responders" together and "non-responders" together in its internal mathematical space before the final classification happens.

Datasets

The study utilizes two distinct datasets to overcome the lack of labels:

  • Medical Segmentation Decathlon (MSD): A public dataset of 281 abdominal CT scans used exclusively to train the Stage I and II segmentation/localization models.
  • In-house Clinical Dataset: 477 CT scans from PDAC patients with treatment response labels based on the RECIST score (a clinical standard for measuring if a tumor is progressing, stable, or regressing). This dataset was split into 420 samples for training and 57 for testing.

Key Insights and Ideas

  • Learning from Unrelated Tasks: The most powerful "aha" moment is that you can improve a classification model (will the tumor shrink?) by pre-training it on a segmentation task (where is the pancreas?), even if the two datasets don't share the same patients.
  • Geometric Assistance: By feeding the segmentation mask into the classifier, the authors essentially gave the AI a "cheat sheet" regarding the tumor's boundaries, allowing the network to focus its "attention" on the most relevant textures.
  • Efficiency Gains: Reusing weights from the segmentation stage wasn't just about accuracy; it led to a 2.5x training speedup and significantly lower energy consumption.
  • Robustness to Imbalance: The authors prioritized the Matthews Correlation Coefficient (MCC) over standard accuracy. In medical imaging, where one class (e.g., non-responders) often outweighs the other, MCC provides a much more honest look at whether the model is actually learning or just guessing the majority class.

Key Results

The incremental improvements of the pipeline are striking:

  • Baseline: A standard 3D ResNet50 performed poorly, with an MCC of only 5.6% (barely better than random).
  • The "Cropping" Effect: Simply narrowing the focus to the pancreatic region (Slice + Informed cropping) jumped the MCC to 18.8%.
  • The Full Pipeline: Adding segmentation forwarding, transfer learning, and triplet loss brought the final MCC to 33.1%.
  • Final Metrics: The complete system achieved an Accuracy of 67.2% and an AUC-ROC of 63.7%. While these numbers might seem modest compared to cat-vs-dog classifiers, they represent a significant step forward for forecasting complex biological responses in PDAC.

Limitations and Future Work

The authors acknowledge that while their method is data-efficient, an AUC-ROC of 63.7% leaves room for improvement before clinical deployment. The test set is also relatively small (57 patients), which can lead to high variance in results.

Future work could involve incorporating multi-modal data (like blood-based tumor markers or genetic data) alongside the CT scans to provide a more holistic view of the patient's condition. The ultimate goal is a tool that helps clinicians choose the specific chemotherapy regimen most likely to work for a specific individual's tumor profile.

Summary updated

Abstract

The prediction of pancreatic ductal adenocarcinoma therapy response is a clinically challenging and important task in this high-mortality tumour entity. The training of neural networks able to tackle this challenge is impeded by a lack of large datasets and the difficult anatomical localisation of the pancreas. Here, we propose a hybrid deep neural network pipeline to predict tumour response to initial chemotherapy which is based on the Response Evaluation Criteria in Solid Tumors (RECIST) score, a standardised method for cancer response evaluation by clinicians as well as tumour markers, and clinical evaluation of the patients. We leverage a combination of representation transfer from segmentation to classification, as well as localisation and representation learning. Our approach yields a remarkably data-efficient method able to predict treatment response with a ROC-AUC of 63.7% using only 477 datasets in total.
BibTeX
@inproceedings{ziller2023exploiting,
  author = {Alexander Ziller and Ayhan Can Erdur and Friederike Jungmann and Daniel Rueckert and Rickmer Braren and Georgios Kaissis},
  title = {{Exploiting Segmentation Labels and Representation Learning to Forecast Therapy Response of PDAC Patients}},
  booktitle = {ISBI},
  year = {2023},
  doi = {10.1109/isbi53787.2023.10230324},
}

Loading related content...

Figure