Step Forward Cross Validation for Bioactivity Prediction: Out of Distribution Validation in Drug Discovery

Udit Surya Saha, Michele Vendruscolo, Anne E. Carpenter, Shantanu Singh, Andreas Bender, Srijit SealbioRxiv 2024 · DOI

TL;DR

This paper introduces a more realistic validation method, "step-forward cross-validation" (SFCV), for machine learning models in drug discovery. It demonstrates that SFCV better reflects real-world drug development scenarios than traditional methods, leading to more reliable predictions of compound bioactivity and a better understanding of model applicability.

What Problem Does This Paper Address?

Drug discovery relies heavily on predicting the bioactivity of new molecules. However, current machine learning models are often validated using methods that don't accurately reflect how these models will be used in practice. Typically, models are trained and tested on data that is very similar, leading to an overestimation of their performance on truly novel, "out-of-distribution" compounds that haven't been synthesized yet. This gap between validation and real-world application can lead to wasted resources and failed drug development efforts. The authors aim to bridge this gap by adapting validation techniques from materials science to drug discovery.

Methods

The core of this work is the introduction and evaluation of k-fold n-step forward cross-validation (SFCV), adapted from materials science. Unlike standard k-fold cross-validation where data is randomly split, SFCV sorts the data (in this case, by logP, a measure of hydrophobicity) and iteratively uses blocks of data for training and testing. This mimics the real-world drug discovery process where compounds are progressively optimized to become more "drug-like" (often involving changes in logP).

They also introduce and utilize two metrics from materials science:

  • Discovery Yield: Measures how well a model predicts compounds with desirable properties (e.g., high potency and low toxicity).
  • Novelty Error: Assesses the model's accuracy on compounds that are structurally distinct from the training data, indicating its ability to generalize to truly new chemical space.

The authors compared SFCV with traditional methods like random split cross-validation and scaffold-based cross-validation.

Datasets

The study used three publicly available datasets for predicting compound bioactivity against specific protein targets:

  • hERG: (1467 compounds) - Inhibition of this target is associated with cardiotoxicity and should be avoided.
  • MAPK14: (1513 compounds) - Inhibition is a positive attribute for treating various diseases.
  • VEGFR2: (1751 compounds) - Inhibition is a therapeutic target for treating angiogenesis-related tumors.

The data consisted of IC50 values, which were converted to pIC50 values for analysis.

Key Insights and Ideas

  • Real-world validation is crucial: Traditional random splitting of data for model validation can be misleading because it doesn't reflect the sequential, optimization-driven nature of drug discovery.
  • SFCV mimics drug discovery: Sorting data by a relevant property like logP and using a step-forward approach in cross-validation better simulates how chemists iteratively design and test compounds.
  • Novelty matters: Simply predicting known compound types well isn't enough; models need to accurately predict novel chemical structures.
  • Discovery Yield and Novelty Error provide deeper insights: These metrics go beyond simple accuracy to assess a model's practical utility in finding promising new drug candidates and its ability to explore new chemical space.

Key Results

  • SFCV is more informative: Sorted SFCV consistently identified more structurally dissimilar compounds in test batches compared to random or scaffold-based splits, indicating its strength in evaluating models for novel structures.
  • Discovery Yield shows promise with SFCV: While discovery yield fluctuated more with SFCV (especially sorted SFCV) compared to random splits, this is attributed to the more challenging, realistic test environment it creates.
  • Novelty Error is more consistent with SFCV: Sorted SFCV demonstrated lower and more consistent novelty error, suggesting it helps minimize overfitting and better assesses generalization to novel compounds.
  • Extrapolation is challenging: Predicting bioactivity for compounds with logP values significantly different from the training data proved difficult for all models, highlighting the need for better extrapolation capabilities.
  • Table 1 Summary: For Random Forest models, CV (Random) generally showed the best R2 and Discovery Yield, but Sorted SFCV was better at selecting novel structures.

Limitations and Future Work

The authors acknowledge that predicting bioactivity for compounds with logP values outside the training data range remains a challenge. They suggest that future work should focus on developing models with better extrapolation capabilities. They also propose extending SFCV to other activity measures beyond IC50, such as inhibition constants (Ki), to gain a more comprehensive understanding of compound-protein interactions. Implementing SFCV within Design-Make-Test-Analyze (DMTA) cycles is also suggested as a way to potentially improve generalization for diverse compounds.

Summary updated

Abstract

Recent advances in machine learning methods for materials science have significantly enhanced accurate predictions of the properties of novel materials. Here, we explore whether these advances can be adapted to drug discovery by addressing the problem of prospective validation - the assessment of the performance of a method on out-of-distribution data. First, we tested whether k-fold n-step forward cross-validation could improve the accuracy of out-of-distribution small molecule bioactivity predictions. We found that it is more helpful than conventional random split cross-validation in describing the accuracy of a model in real-world drug discovery settings. We also analyzed discovery yield and novelty error, finding that these two metrics provide an understanding of the applicability domain of models and an assessment of their ability to predict molecules with desirable bioactivity compared to other small molecules. Based on these results, we recommend incorporating a k-fold n-step forward cross-validation and these metrics when building state-of-the-art models for bioactivity prediction in drug discovery.
BibTeX
@misc{saha2024step,
  author = {Udit Surya Saha and Michele Vendruscolo and Anne E. Carpenter and Shantanu Singh and Andreas Bender and Srijit Seal},
  title = {{Step Forward Cross Validation for Bioactivity Prediction: Out of Distribution Validation in Drug Discovery}},
  year = {2024},
  doi = {10.1101/2024.07.02.601740},
  howpublished = {bioRxiv},
}

Loading related content...

Figure