TL;DR
This paper introduces a more realistic validation method, "step-forward cross-validation" (SFCV), for machine learning models in drug discovery. It demonstrates that SFCV better reflects real-world drug development scenarios than traditional methods, leading to more reliable predictions of compound bioactivity and a better understanding of model applicability.
What Problem Does This Paper Address?
Drug discovery relies heavily on predicting the bioactivity of new molecules. However, current machine learning models are often validated using methods that don't accurately reflect how these models will be used in practice. Typically, models are trained and tested on data that is very similar, leading to an overestimation of their performance on truly novel, "out-of-distribution" compounds that haven't been synthesized yet. This gap between validation and real-world application can lead to wasted resources and failed drug development efforts. The authors aim to bridge this gap by adapting validation techniques from materials science to drug discovery.
Methods
The core of this work is the introduction and evaluation of k-fold n-step forward cross-validation (SFCV), adapted from materials science. Unlike standard k-fold cross-validation where data is randomly split, SFCV sorts the data (in this case, by logP, a measure of hydrophobicity) and iteratively uses blocks of data for training and testing. This mimics the real-world drug discovery process where compounds are progressively optimized to become more "drug-like" (often involving changes in logP).
They also introduce and utilize two metrics from materials science:
- Discovery Yield: Measures how well a model predicts compounds with desirable properties (e.g., high potency and low toxicity).
- Novelty Error: Assesses the model's accuracy on compounds that are structurally distinct from the training data, indicating its ability to generalize to truly new chemical space.
The authors compared SFCV with traditional methods like random split cross-validation and scaffold-based cross-validation.
Datasets
The study used three publicly available datasets for predicting compound bioactivity against specific protein targets:
- hERG: (1467 compounds) - Inhibition of this target is associated with cardiotoxicity and should be avoided.
- MAPK14: (1513 compounds) - Inhibition is a positive attribute for treating various diseases.
- VEGFR2: (1751 compounds) - Inhibition is a therapeutic target for treating angiogenesis-related tumors.
The data consisted of IC50 values, which were converted to pIC50 values for analysis.
Key Insights and Ideas
- Real-world validation is crucial: Traditional random splitting of data for model validation can be misleading because it doesn't reflect the sequential, optimization-driven nature of drug discovery.
- SFCV mimics drug discovery: Sorting data by a relevant property like logP and using a step-forward approach in cross-validation better simulates how chemists iteratively design and test compounds.
- Novelty matters: Simply predicting known compound types well isn't enough; models need to accurately predict novel chemical structures.
- Discovery Yield and Novelty Error provide deeper insights: These metrics go beyond simple accuracy to assess a model's practical utility in finding promising new drug candidates and its ability to explore new chemical space.
Key Results
- SFCV is more informative: Sorted SFCV consistently identified more structurally dissimilar compounds in test batches compared to random or scaffold-based splits, indicating its strength in evaluating models for novel structures.
- Discovery Yield shows promise with SFCV: While discovery yield fluctuated more with SFCV (especially sorted SFCV) compared to random splits, this is attributed to the more challenging, realistic test environment it creates.
- Novelty Error is more consistent with SFCV: Sorted SFCV demonstrated lower and more consistent novelty error, suggesting it helps minimize overfitting and better assesses generalization to novel compounds.
- Extrapolation is challenging: Predicting bioactivity for compounds with logP values significantly different from the training data proved difficult for all models, highlighting the need for better extrapolation capabilities.
- Table 1 Summary: For Random Forest models, CV (Random) generally showed the best R2 and Discovery Yield, but Sorted SFCV was better at selecting novel structures.
Limitations and Future Work
The authors acknowledge that predicting bioactivity for compounds with logP values outside the training data range remains a challenge. They suggest that future work should focus on developing models with better extrapolation capabilities. They also propose extending SFCV to other activity measures beyond IC50, such as inhibition constants (Ki), to gain a more comprehensive understanding of compound-protein interactions. Implementing SFCV within Design-Make-Test-Analyze (DMTA) cycles is also suggested as a way to potentially improve generalization for diverse compounds.
Summary updated