Learning curves for drug response prediction in cancer cell lines

Alexander Partin, Thomas S. Brettin, Yvonne A. Evrard, Yitan Zhu, Hyun Seung Yoo, Fangfang Xia, Songhao Jiang, Austin Clyde, Maulik Shukla, Michael Fonstein, James H. Doroshow, Rick L. Stevens

Abstract

Motivated by the size of cell line drug sensitivity data, researchers have been developing machine learning (ML) models for predicting drug response to advance cancer treatment. As drug sensitivity studies continue generating data, a common question is whether the proposed predictors can further improve the generalization performance with more training data. We utilize empirical learning curves for evaluating and comparing the data scaling properties of two neural networks (NNs) and two gradient boosting decision tree (GBDT) models trained on four drug screening datasets. The learning curves are accurately fitted to a power law model, providing a framework for assessing the data scaling behavior of these predictors. The curves demonstrate that no single model dominates in terms of prediction performance across all datasets and training sizes, suggesting that the shape of these curves depends on the unique model-dataset pair. The multi-input NN (mNN), in which gene expressions and molecular drug descriptors are input into separate subnetworks, outperforms a single-input NN (sNN). In contrast, a GBDT with hyperparameter tuning exhibits superior performance at the lower range of training sizes for two datasets, whereas the mNN performs better at the higher range. The trajectory of the curves suggests that increasing the sample size is expected to further improve prediction scores of both NNs. These observations demonstrate the benefit of using learning curves to evaluate predictors, providing a broader perspective on the overall data scaling characteristics.

Introduction

In the quest for precision oncology, researchers use machine learning (ML) to predict drug sensitivity based on the genomic profiles of cancer cell lines. As pharmacogenomic datasets grow, a critical question arises: Are we hitting a plateau, or will more data actually make our models better?

Most studies evaluate models using a single performance metric on a fixed dataset. However, this "snapshot" approach fails to reveal how a model evolves as it consumes more information. This paper addresses this by using learning curves—plots of model performance versus training set size—to provide a "forward-looking" metric that helps determine the value of future data collection and the inherent learning capacity of different architectures.

Methods

The authors employed a rigorous framework to generate and analyze learning curves. They fitted the generalization score $s$ as a function of training size $m$ using the power law equation $s(m) = am^b + c$, where $b$ is the scaling exponent and $c$ represents the irreducible error.

Empirical learning curves showing (a) linear scale and (b) log2 scale, illustrating the small-data, power-law, and irreducible error regions.

The study compared four primary model configurations:

  • sNN (Single-input Neural Network): A standard deep learning model where cell and drug features are concatenated at the start.
  • mNN (Multi-input Neural Network): A specialized architecture where gene expressions and drug descriptors are processed by separate subnetworks before being merged.
  • GBDT (Gradient Boosting Decision Trees): Tested in both default (dGBDT) and hyperparameter-tuned (hGBDT) versions.

To ensure robust results, the team used 20 random data splits and a sequential sampling strategy to mimic the temporal growth of a research study, leveraging High-Performance Computing (HPC) to handle the massive computational load.

Datasets

The study utilized four major public drug screening benchmarks: GDSC1, GDSC2, CTRP v2, and NCI-60. For features, they used RNA-Seq data (976 "landmark" genes) to represent cell lines and Mordred molecular descriptors (1,613 features) to represent drug chemistry.

Distribution of dose-independent drug response (AUC values) across the four primary datasets used in the study.

Key Results and Insights

  • No Universal Winner: The "best" model depends entirely on the dataset and the amount of data available. A model that wins at 10,000 samples might lose at 100,000 samples.
  • The Architecture Advantage: The mNN consistently outperformed the simpler sNN. This suggests that allowing the model to learn independent representations for biological (cell) and chemical (drug) features before combining them is a superior strategy for this domain.
  • GBDTs as "Sprint" Champions: Hyperparameter-tuned GBDTs (hGBDT) often showed superior performance in the "small-data" region but tended to plateau earlier than Neural Networks.
  • Forecasting the Future: Because the learning curves fit the power law so accurately ($R^2 > 0.99$), researchers can mathematically estimate how much more data is required to reach a specific accuracy threshold. For example, doubling the training size for the mNN on the CTRP dataset would reduce the error by roughly 15%.
  • Scaling Trajectory: The trajectories for Neural Networks on most datasets showed no signs of flattening out in the power-law region, implying that the field has not yet reached "peak data" and further investment in data generation is justified.

Conclusion

This study demonstrates the benefit of using learning curves to evaluate predictors, providing a broader perspective on overall data scaling characteristics. The fitted power law curves provide a forward-looking performance metric and can serve as a co-design tool to guide experimental biologists and computational scientists in the design of future experiments, particularly in data-scarce environments like Patient-Derived Xenografts (PDXs) and organoids.

Summary updated

Abstract

Background Motivated by the size and availability of cell line drug sensitivity data, researchers have been developing machine learning (ML) models for predicting drug response to advance cancer treatment. As drug sensitivity studies continue generating drug response data, a common question is whether the generalization performance of existing prediction models can be further improved with more training data. Methods We utilize empirical learning curves for evaluating and comparing the data scaling properties of two neural networks (NNs) and two gradient boosting decision tree (GBDT) models trained on four cell line drug screening datasets. The learning curves are accurately fitted to a power law model, providing a framework for assessing the data scaling behavior of these models. Results The curves demonstrate that no single model dominates in terms of prediction performance across all datasets and training sizes, thus suggesting that the actual shape of these curves depends on the unique pair of an ML model and a dataset. The multi-input NN (mNN), in which gene expressions of cancer cells and molecular drug descriptors are input into separate subnetworks, outperforms a single-input NN (sNN), where the cell and drug features are concatenated for the input layer. In contrast, a GBDT with hyperparameter tuning exhibits superior performance as compared with both NNs at the lower range of training set sizes for two of the tested datasets, whereas the mNN consistently performs better at the higher range of training sizes. Moreover, the trajectory of the curves suggests that increasing the sample size is expected to further improve prediction scores of both NNs. These observations demonstrate the benefit of using learning curves to evaluate prediction models, providing a broader perspective on the overall data scaling characteristics. Conclusions A fitted power law learning curve provides a forward-looking metric for analyzing prediction performance and can serve as a co-design tool to guide experimental biologists and computational scientists in the design of future experiments in prospective research studies.
BibTeX
@article{partin2021learning,
  author = {Alexander Partin and Thomas S. Brettin and Yvonne A. Evrard and Yitan Zhu and Hyun Seung Yoo and Fangfang Xia and Songhao Jiang and Austin Clyde and Maulik Shukla and Michael Fonstein and James H. Doroshow and Rick L. Stevens},
  title = {{Learning curves for drug response prediction in cancer cell lines}},
  journal = {BMC Bioinformatics},
  year = {2021},
  doi = {10.1186/s12859-021-04163-y},
}
Figure