Exposing the Limitations of Molecular Machine Learning with Activity Cliffs

Derek van Tilborg, Alisa Alenicheva, Francesca GrisoniJournal of Chemical Information and Modeling 2022 · OpenReview

TL;DR

This paper investigates how well machine learning (ML) models predict the activity of "activity cliffs" – molecules that are structurally very similar but have vastly different biological effects. They found that traditional ML methods using molecular descriptors generally outperform complex deep learning models on these challenging cases, highlighting a critical blind spot in current ML for drug discovery.

What Problem Does This Paper Address?

Machine learning (ML) has become indispensable in drug discovery, excelling at predicting molecular properties like bioactivity. However, a significant challenge arises from "activity cliffs": pairs of molecules that are almost identical structurally but exhibit dramatically different potencies. These cliffs are crucial for understanding structure-activity relationships and for optimizing drug candidates. Despite their importance, activity cliffs have been largely overlooked in the evaluation of ML models. This means that models with high overall accuracy might fail spectacularly on these critical edge cases, hindering their real-world applicability. This paper aims to systematically benchmark various ML and deep learning (DL) approaches specifically on their ability to handle activity cliffs, identifying current limitations and suggesting future directions.

Methods

The researchers employed a comprehensive benchmarking approach, evaluating 24 different ML and DL methods.

  • Traditional ML: Four established algorithms were used: K-Nearest Neighbors (KNN), Random Forest (RF), Gradient Boosting Machine (GBM), and Support Vector Regression (SVM). These were combined with four types of molecular descriptors: Extended Connectivity Fingerprints (ECFPs), MACCS keys, Weighted Holistic Invariant Molecular (WHIM) descriptors, and physicochemical properties.
  • Deep Learning:
    • Graph-based: Four graph neural network architectures were tested: Message Passing Neural Network (MPNN), Graph Attention Network (GAT), Graph Convolutional Network (GCN), and Attentive Fingerprint (AFP). These learn directly from molecular graph representations.
  • SMILES-based: Three models were used that learn from SMILES strings: Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Transformers.
  • Activity Cliff Definition: Molecules were considered "highly similar" if they met a similarity threshold (>= 0.9) across at least one of three metrics: substructure similarity (ECFP Tanimoto), scaffold similarity (ECFP Tanimoto on molecular cores), or SMILES string similarity (Levenshtein distance). Activity cliffs were then identified as pairs of highly similar molecules with at least a tenfold difference in reported potency.
  • Data Splitting: A novel strategy was employed to ensure proportional representation of activity cliff compounds in training and test sets while preserving structural similarity, using spectral clustering and stratified random sampling.
  • Performance Metrics: The primary metrics were Root-Mean-Square Error (RMSE) for overall performance and RMSEcliff for performance specifically on activity cliff compounds.

Datasets

The study utilized curated bioactivity data from 30 macromolecular targets, sourced from ChEMBL v29. These datasets varied in size, from small (615 molecules for JAK1) to large (3657 molecules for DRD3), and covered diverse target families relevant to drug discovery (e.g., kinases, nuclear receptors). The authors emphasize their rigorous data curation process to ensure data quality. The study also introduced MoleculeACE, an open-access benchmarking platform that includes these datasets and the developed metrics for assessing ML performance on activity cliffs.

Key Insights and Ideas

  • Activity Cliffs are a Major Hurdle: All ML and DL methods struggled to accurately predict the activity of molecules within activity cliffs, demonstrating a consistent weakness across the board.
  • Descriptors Trump Complexity (for Cliffs): Traditional ML methods that rely on human-engineered molecular descriptors generally outperformed more complex deep learning approaches when predicting activity cliff behavior. This suggests that current DL methods, while powerful, may not be effectively capturing the subtle structural nuances that lead to activity cliffs.
  • Case-by-Case Variability is High: Performance on activity cliffs is highly dependent on the specific dataset and target. A model that performs well overall might still fail significantly on certain activity cliffs, underscoring the need for cliff-specific evaluation.
  • Overall RMSE Can Be Misleading: A low overall RMSE doesn't guarantee good performance on activity cliffs. Models with similar overall performance can show vastly different abilities to handle these challenging cases.
  • Low-Data Regimes are Particularly Challenging: While not a direct correlation, the study suggests that the problem of predicting activity cliffs is exacerbated in low-data scenarios, a common situation in drug discovery.

Key Results

  • Descriptor Performance: ECFPs generally yielded the lowest average prediction error on activity cliff compounds compared to other descriptors. Non-binary descriptors (WHIM, physicochemical) performed worse.
  • Traditional ML vs. Deep Learning: SVM coupled with ECFPs emerged as the best-performing traditional ML method on average for activity cliffs. Deep learning methods, particularly graph-based ones, generally struggled more than descriptor-based methods. LSTM models showed the best performance among the deep learning approaches.
  • RMSEcliff vs. RMSE: While RMSEcliff and RMSE were generally correlated (r=0.81 on average), activity cliff compounds could lead to overestimated overall model performance by up to 0.54 log units when only RMSE was considered.
  • MoleculeACE Platform: The study's methods, metrics, and results are encapsulated in the MoleculeACE platform, providing a standardized tool for evaluating ML models on activity cliffs.

Limitations and Future Work

  • Mechanistic Understanding: The study identified the performance gap but did not elucidate the precise mechanistic reasons why current molecular representations and DL algorithms fail to capture activity cliff information.
  • Data Quality: Despite rigorous curation, the authors acknowledge that public bioactivity data may contain undetectable experimental noise, which could influence results.
  • Future Directions:
    • Novel Algorithms: Development of new algorithms specifically designed to capture structure-activity "discontinuities" is crucial.
  • Structure-Based DL: Incorporating structural information of the macromolecular target alongside ligand information in deep learning models is a promising avenue, though consensus on its benefit is still lacking.
  • Low-Data Scenarios: More efficient ML/DL models for low-data regimes are needed.
  • MoleculeACE Expansion: The MoleculeACE platform is intended to be expanded and used by the community to drive progress in this area, potentially with custom thresholds for similarity and potency differences.

Summary updated

Abstract

Machine learning has become a crucial tool in drug discovery and chemistry at large, e.g., to predict molecular properties, such as bioactivity, with high accuracy. However, activity cliffs─pairs of molecules that are highly similar in their structure but exhibit large differences in potency─have received limited attention for their effect on model performance. Not only are these edge cases informative for molecule discovery and optimization but also models that are well equipped to accurately predict the potency of activity cliffs have increased potential for prospective applications. Our work aims to fill the current knowledge gap on best-practice machine learning methods in the presence of activity cliffs. We benchmarked a total of 24 machine and deep learning approaches on curated bioactivity data from 30 macromolecular targets for their performance on activity cliff compounds. While all methods struggled in the presence of activity cliffs, machine learning approaches based on molecular descriptors outperformed more complex deep learning methods. Our findings highlight large case-by-case differences in performance, advocating for (a) the inclusion of dedicated “activity-cliff-centered” metrics during model development and evaluation and (b) the development of novel algorithms to better predict the properties of activity cliffs. To this end, the methods, metrics, and results of this study have been encapsulated into an open-access benchmarking platform named MoleculeACE (Activity Cliff Estimation, available on GitHub at: https://github.com/molML/MoleculeACE). MoleculeACE is designed to steer the community toward addressing the pressing but overlooked limitation of molecular machine learning models posed by activity cliffs.
BibTeX
@article{tilborg2022exposing,
  author = {Derek van Tilborg and Alisa Alenicheva and Francesca Grisoni},
  title = {{Exposing the Limitations of Molecular Machine Learning with Activity Cliffs}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2022},
  url = {https://openreview.net/forum?id=5C5yUsoI8b},
}
Figure