TL;DR
This paper investigates how well machine learning (ML) models predict the activity of "activity cliffs" – molecules that are structurally very similar but have vastly different biological effects. They found that traditional ML methods using molecular descriptors generally outperform complex deep learning models on these challenging cases, highlighting a critical blind spot in current ML for drug discovery.
What Problem Does This Paper Address?
Machine learning (ML) has become indispensable in drug discovery, excelling at predicting molecular properties like bioactivity. However, a significant challenge arises from "activity cliffs": pairs of molecules that are almost identical structurally but exhibit dramatically different potencies. These cliffs are crucial for understanding structure-activity relationships and for optimizing drug candidates. Despite their importance, activity cliffs have been largely overlooked in the evaluation of ML models. This means that models with high overall accuracy might fail spectacularly on these critical edge cases, hindering their real-world applicability. This paper aims to systematically benchmark various ML and deep learning (DL) approaches specifically on their ability to handle activity cliffs, identifying current limitations and suggesting future directions.
Methods
The researchers employed a comprehensive benchmarking approach, evaluating 24 different ML and DL methods.
- Traditional ML: Four established algorithms were used: K-Nearest Neighbors (KNN), Random Forest (RF), Gradient Boosting Machine (GBM), and Support Vector Regression (SVM). These were combined with four types of molecular descriptors: Extended Connectivity Fingerprints (ECFPs), MACCS keys, Weighted Holistic Invariant Molecular (WHIM) descriptors, and physicochemical properties.
- Deep Learning:
- Graph-based: Four graph neural network architectures were tested: Message Passing Neural Network (MPNN), Graph Attention Network (GAT), Graph Convolutional Network (GCN), and Attentive Fingerprint (AFP). These learn directly from molecular graph representations.
- SMILES-based: Three models were used that learn from SMILES strings: Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Transformers.
- Activity Cliff Definition: Molecules were considered "highly similar" if they met a similarity threshold (>= 0.9) across at least one of three metrics: substructure similarity (ECFP Tanimoto), scaffold similarity (ECFP Tanimoto on molecular cores), or SMILES string similarity (Levenshtein distance). Activity cliffs were then identified as pairs of highly similar molecules with at least a tenfold difference in reported potency.
- Data Splitting: A novel strategy was employed to ensure proportional representation of activity cliff compounds in training and test sets while preserving structural similarity, using spectral clustering and stratified random sampling.
- Performance Metrics: The primary metrics were Root-Mean-Square Error (RMSE) for overall performance and RMSEcliff for performance specifically on activity cliff compounds.
Datasets
The study utilized curated bioactivity data from 30 macromolecular targets, sourced from ChEMBL v29. These datasets varied in size, from small (615 molecules for JAK1) to large (3657 molecules for DRD3), and covered diverse target families relevant to drug discovery (e.g., kinases, nuclear receptors). The authors emphasize their rigorous data curation process to ensure data quality. The study also introduced MoleculeACE, an open-access benchmarking platform that includes these datasets and the developed metrics for assessing ML performance on activity cliffs.
Key Insights and Ideas
- Activity Cliffs are a Major Hurdle: All ML and DL methods struggled to accurately predict the activity of molecules within activity cliffs, demonstrating a consistent weakness across the board.
- Descriptors Trump Complexity (for Cliffs): Traditional ML methods that rely on human-engineered molecular descriptors generally outperformed more complex deep learning approaches when predicting activity cliff behavior. This suggests that current DL methods, while powerful, may not be effectively capturing the subtle structural nuances that lead to activity cliffs.
- Case-by-Case Variability is High: Performance on activity cliffs is highly dependent on the specific dataset and target. A model that performs well overall might still fail significantly on certain activity cliffs, underscoring the need for cliff-specific evaluation.
- Overall RMSE Can Be Misleading: A low overall RMSE doesn't guarantee good performance on activity cliffs. Models with similar overall performance can show vastly different abilities to handle these challenging cases.
- Low-Data Regimes are Particularly Challenging: While not a direct correlation, the study suggests that the problem of predicting activity cliffs is exacerbated in low-data scenarios, a common situation in drug discovery.
Key Results
- Descriptor Performance: ECFPs generally yielded the lowest average prediction error on activity cliff compounds compared to other descriptors. Non-binary descriptors (WHIM, physicochemical) performed worse.
- Traditional ML vs. Deep Learning: SVM coupled with ECFPs emerged as the best-performing traditional ML method on average for activity cliffs. Deep learning methods, particularly graph-based ones, generally struggled more than descriptor-based methods. LSTM models showed the best performance among the deep learning approaches.
- RMSEcliff vs. RMSE: While RMSEcliff and RMSE were generally correlated (r=0.81 on average), activity cliff compounds could lead to overestimated overall model performance by up to 0.54 log units when only RMSE was considered.
- MoleculeACE Platform: The study's methods, metrics, and results are encapsulated in the MoleculeACE platform, providing a standardized tool for evaluating ML models on activity cliffs.
Limitations and Future Work
- Mechanistic Understanding: The study identified the performance gap but did not elucidate the precise mechanistic reasons why current molecular representations and DL algorithms fail to capture activity cliff information.
- Data Quality: Despite rigorous curation, the authors acknowledge that public bioactivity data may contain undetectable experimental noise, which could influence results.
- Future Directions:
- Novel Algorithms: Development of new algorithms specifically designed to capture structure-activity "discontinuities" is crucial.
- Structure-Based DL: Incorporating structural information of the macromolecular target alongside ligand information in deep learning models is a promising avenue, though consensus on its benefit is still lacking.
- Low-Data Scenarios: More efficient ML/DL models for low-data regimes are needed.
- MoleculeACE Expansion: The MoleculeACE platform is intended to be expanded and used by the community to drive progress in this area, potentially with custom thresholds for similarity and potency differences.
Summary updated