Learning curves for drug response prediction in cancer cell lines

Alexander Partin, Thomas S. Brettin, Yvonne A. Evrard, Yitan Zhu, Hyun Seung Yoo, Fangfang Xia, Songhao Jiang, Austin Clyde, Maulik Shukla, Michael Fonstein, James H. Doroshow, Rick L. Stevens

Introduction

In the field of personalized oncology, machine learning (ML) models are increasingly used to predict how specific cancer cell lines will respond to various therapeutic agents. As pharmacogenomic datasets continue to expand, researchers face a fundamental question: at what point does collecting more data yield diminishing returns? Most current research provides only a static "snapshot" of model performance on a fixed dataset, failing to reveal whether a model has reached its theoretical limit or if further data acquisition would significantly enhance predictive accuracy.

This study addresses this gap by employing learning curves—plots of model performance against training set size—to characterize the data scaling behavior of different algorithms. By fitting these curves to empirical data, the researchers can forecast future performance and compare model architectures across various scales.

Methodology

The researchers developed a systematic workflow to generate and fit learning curves using a power law expression:

$$s(m) = am^b + c$$

In this equation, $s(m)$ represents the generalization score (Mean Absolute Error), $m$ is the training set size, and the parameters $a, b, c$ represent the scaling exponent and the irreducible error.

Models Evaluated

The study compared two primary classes of models:

  1. Neural Networks (NNs): Specifically, a single-input NN (sNN), which concatenates cell and drug features at the input layer, and a multi-input NN (mNN), which utilizes separate subnetworks to encode biological and chemical features before merging them.
  2. Gradient Boosting Decision Trees (GBDT): Both a default version (dGBDT) and a hyperparameter-tuned version (hGBDT) were tested.

To ensure statistical robustness, the team utilized 20 random data splits and 10 different training subset sizes per split, leveraging High-Performance Computing (HPC) to manage the intensive training requirements.

The Three Regions of Learning

A central finding of the study is the empirical validation of the three distinct regions of model performance as data scales.

Learning curves on linear and log2 scales showing the three distinct regions of model performance.

As illustrated in the figure above, the learning process follows a predictable trajectory:

  • Small-data Region: Performance is often poor and akin to random guessing.
  • Power-law Region: This is the "sweet spot" where performance improves steadily and predictably as more data is added.
  • Irreducible Error Region: Eventually, the curve plateaus as the model reaches the limit of what can be learned from the available features, representing the noise or inherent complexity in the biological data.

Key Insights and Results

The "Model-Dataset Pair" Principle

The research highlights that no single model is universally superior. The shape and trajectory of a learning curve are unique properties of the specific combination of an algorithm and a dataset. However, certain architectural trends emerged:

  • Architectural Superiority of mNN: The multi-input architecture consistently outperformed the single-input approach. By learning distinct representations for cell line genomics and drug chemistry separately before integration, the mNN scales more effectively.
  • The Overtake Point: While hyperparameter-tuned GBDTs often showed better performance when data was scarce, the NNs (particularly mNN) exhibited steeper scaling trajectories. As the training size increased, the NNs eventually overtook the GBDTs in accuracy.
  • Quantitative Gains: Optimized models demonstrated improvements of 8% to 42% over baseline default models. On the CTRP dataset, the researchers calculated that doubling the training size would reduce the mNN error by approximately 15%.

Conclusion and Future Directions

By confirming that drug response prediction follows power law scaling ($R^2 > 0.99$), this study provides a framework for researchers to estimate the amount of data required to reach specific accuracy thresholds. While cell lines are imperfect proxies for human tumors, the methodology established here paves the way for analyzing more complex models, such as Patient-Derived Xenografts (PDXs) or Organoids, as those datasets grow. Ultimately, understanding these scaling laws allows for more strategic resource allocation in the development of precision medicine tools.

Summary updated

Abstract

Background Motivated by the size and availability of cell line drug sensitivity data, researchers have been developing machine learning (ML) models for predicting drug response to advance cancer treatment. As drug sensitivity studies continue generating drug response data, a common question is whether the generalization performance of existing prediction models can be further improved with more training data. Methods We utilize empirical learning curves for evaluating and comparing the data scaling properties of two neural networks (NNs) and two gradient boosting decision tree (GBDT) models trained on four cell line drug screening datasets. The learning curves are accurately fitted to a power law model, providing a framework for assessing the data scaling behavior of these models. Results The curves demonstrate that no single model dominates in terms of prediction performance across all datasets and training sizes, thus suggesting that the actual shape of these curves depends on the unique pair of an ML model and a dataset. The multi-input NN (mNN), in which gene expressions of cancer cells and molecular drug descriptors are input into separate subnetworks, outperforms a single-input NN (sNN), where the cell and drug features are concatenated for the input layer. In contrast, a GBDT with hyperparameter tuning exhibits superior performance as compared with both NNs at the lower range of training set sizes for two of the tested datasets, whereas the mNN consistently performs better at the higher range of training sizes. Moreover, the trajectory of the curves suggests that increasing the sample size is expected to further improve prediction scores of both NNs. These observations demonstrate the benefit of using learning curves to evaluate prediction models, providing a broader perspective on the overall data scaling characteristics. Conclusions A fitted power law learning curve provides a forward-looking metric for analyzing prediction performance and can serve as a co-design tool to guide experimental biologists and computational scientists in the design of future experiments in prospective research studies.
BibTeX
@article{partin2021learning,
  author = {Alexander Partin and Thomas S. Brettin and Yvonne A. Evrard and Yitan Zhu and Hyun Seung Yoo and Fangfang Xia and Songhao Jiang and Austin Clyde and Maulik Shukla and Michael Fonstein and James H. Doroshow and Rick L. Stevens},
  title = {{Learning curves for drug response prediction in cancer cell lines}},
  journal = {BMC Bioinformatics},
  year = {2021},
  doi = {10.1186/s12859-021-04163-y},
}
Figure