Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images

Alexander Partin, Thomas Brettin, Yitan Zhu, James M. Dolezal, Sara Kochanny, Alexander T. Pearson, Maulik Shukla, Yvonne A. Evrard, James H. Doroshow, Rick Stevens

TL;DR

Researchers have developed MM-Net, a multimodal deep learning framework that predicts how tumors in Patient-Derived Xenografts (PDXs) will respond to drugs by combining gene expression data with histology images. By using clever data augmentation—such as treating single drugs as "pseudo-pairs" and flipping the order of drug combinations—the team significantly improved prediction accuracy despite the chronic data scarcity that plagues PDX research.

What Problem Does This Paper Address?

In precision oncology, we want to know if a drug will work before giving it to a patient. Historically, researchers used Cancer Cell Lines (CCLs) for this, but CCLs are grown in petri dishes and don't mimic the complex "neighborhood" of a real tumor. Patient-Derived Xenografts (PDXs)—human tumors grown in mice—are much better models because they preserve the tumor’s structural and genetic diversity.

The catch? PDX data is incredibly expensive and time-consuming to generate. While CCL datasets have hundreds of thousands of samples, PDX datasets are orders of magnitude smaller. This "small data" problem makes it hard to train robust deep learning models. Furthermore, most existing models rely solely on genomic data, ignoring the rich spatial information hidden in histology slides (microscope images of tumor tissue).

Methods

The authors introduced two main innovations to tackle these challenges:

  1. MM-Net (Multimodal Neural Network): This architecture uses "late integration." It features separate "specialist" subnetworks to process different data types:
  2. Gene Expression (GE): Dense layers to process 942 landmark genes.
  3. Histology Images: A pre-trained Xception convolutional neural network (CNN) to extract features from whole-slide image tiles.
  4. Drug Descriptors: Numerical representations of chemical structures. The outputs of these specialists are then merged (concatenated) into a final set of layers to predict a binary response: "Success" (tumor regression) or "Failure."

  5. Symmetry-Based Data Augmentation: To beef up their small dataset, the authors used two tricks:

  6. Homogenization: They treated single-drug treatments as pairs of the same drug (e.g., Drug A becomes Drug A + Drug A). This allowed them to use one consistent model architecture for everything.
  7. Drug-Pair Flipping: Since the order of drugs in a cocktail shouldn't change the biological outcome, they doubled their drug-pair data by simply switching the positions of the two drugs in the input (e.g., [A, B] and [B, A]).

Datasets

The study utilized unpublished data from the NCI Patient-Derived Models Repository (PDMR).

  • Scope: 96 PDX models derived from 89 unique patients.
  • Features: 487 unique gene expression profiles and 487 histology whole-slide images (WSIs), which were broken down into over 177,000 smaller image tiles.
  • Treatments: 12 single-drug and 36 drug-pair treatment arms, resulting in 6,962 total samples after augmentation.

Key Insights and Ideas

  • Images Add Context: While gene expression tells you what proteins might be made, histology images show you how the tumor is organized. The study confirms that adding this spatial "vision" to the genetic "blueprint" improves the model's ability to generalize.
  • The Power of Symmetry: The most effective "free" way to improve performance wasn't a more complex model, but rather the data augmentation. By teaching the model that the order of drugs in a combination is irrelevant, the researchers effectively doubled their most valuable data points.
  • Late Integration is Key: By encoding each modality separately before merging them, the model prevents one high-dimensional feature set (like thousands of image pixels) from "overpowering" another (like a few hundred gene values) during the early stages of learning.

Key Results

The researchers benchmarked MM-Net against several baselines, including LightGBM and unimodal neural networks (using only GE or only images).

  • Augmentation Wins: The neural network using the full augmented dataset significantly outperformed versions that ignored the "flipped" drug pairs (p-value < 0.01).
  • Multimodal Superiority: MM-Net achieved the highest Matthews Correlation Coefficient (MCC) of 0.3102, statistically outperforming the GE-only model (0.2958) and the image-only model (0.2124).
  • Robustness: While MM-Net didn't always win on every single metric (like AUROC, where it was neck-and-neck with LightGBM), it was the most consistent performer across different data splits when measured by MCC, which is a more reliable metric for imbalanced datasets.

Limitations and Future Work

Despite the gains, the authors noted a high degree of variance across data splits. In about half of the test scenarios, the models struggled to learn a meaningful mapping, likely due to the inherent noise and extreme heterogeneity of the PDX samples.

Future work will likely focus on:

  • Attention Mechanisms: Instead of just taking the average of image tiles, using attention to let the model "zoom in" on the most relevant parts of a histology slide.
  • Feature Selection: Reducing the dimensionality of the genomic data to prevent overfitting.
  • Broader Modalities: Incorporating other "omics" data, like copy-number alterations (CNAs) or proteomics, to provide an even more holistic view of the tumor.

Summary updated

Abstract

Patient-derived xenografts (PDXs) are an appealing platform for preclinical drug studies because the in vivo environment of PDXs helps preserve tumor heterogeneity and usually better mimics drug response of patients with cancer compared to CCLs. We investigate multimodal neural network (MM-Net) and data augmentation for drug response prediction in PDXs. The MM-Net learns to predict response using drug descriptors, gene expressions (GE), and histology whole-slide images (WSIs) where the multi-modality refers to the tumor features. We explore whether the integration of WSIs with GE improves predictions as compared with models that use GE alone. We use two methods to address the limited number of response values: 1) homogenize drug representations which allows to combine single-drug and drug-pairs treatments into a single dataset, 2) augment drug-pair samples by switching the order of drug features which doubles the sample size of all drug-pair samples. These methods enable us to combine single-drug and drug-pair treatments, allowing us to train multimodal and unimodal neural networks (NNs) without changing architectures or the dataset. Prediction performance of three unimodal NNs which use GE are compared to assess the contribution of data augmentation methods. NN that uses the full dataset which includes the original and the augmented drug-pair treatments as well as single-drug treatments significantly outperforms NNs that ignore either the augmented drug-pairs or the single-drug treatments. In assessing the contribution of multimodal learning based on the MCC metric, MM-Net statistically significantly outperforms all the baselines. Our results show that data augmentation and integration of histology images with GE can improve prediction performance of drug response in PDXs.
BibTeX
@inproceedings{partin2023data,
  author = {Alexander Partin and Thomas Brettin and Yitan Zhu and James M. Dolezal and Sara Kochanny and Alexander T. Pearson and Maulik Shukla and Yvonne A. Evrard and James H. Doroshow and Rick Stevens},
  title = {{Data augmentation and multimodal learning for predicting drug response in patient-derived xenografts from gene expressions and histology images}},
  booktitle = {Frontiers in Medicine},
  year = {2023},
  doi = {10.3389/fmed.2023.1058919},
}
Figure