Correction to "Exposing the Limitations of Molecular Machine Learning with Activity Cliffs"

Derek van Tilborg, Alisa Alenicheva, Francesca Grisoni

TL;DR

This paper is a correction to a previous study that investigated the limitations of molecular machine learning (ML) models when dealing with "activity cliffs" – subtle molecular changes leading to significant differences in biological activity. The correction addresses a software bug that affected data splitting, leading to a retraining of models and updated figures and tables, confirming the original conclusions.

What Problem Does This Paper Address?

The original paper aimed to expose the limitations of molecular machine learning models, particularly when applied to datasets containing "activity cliffs." Activity cliffs are a critical challenge in drug discovery, as they represent pairs of structurally similar molecules with vastly different biological activities. Accurately predicting these differences is crucial for designing effective drugs. The authors found that existing ML models struggled to generalize to these challenging cases, often performing poorly on activity cliff compounds compared to the overall dataset. This correction addresses an issue in the original study's methodology: a software bug in how training and testing sets were split, which incorrectly labeled some activity cliff pairs. This necessitated a re-evaluation of the results.

Methods

The core methodology involves training and evaluating various machine learning models on a curated set of chemical and biological data. The models used include traditional machine learning algorithms like Gradient Boosting Machines (GBM), Support Vector Machines (SVM), Random Forests (RF), and k-Nearest Neighbors (KNN), as well as deep learning architectures such as Graph Neural Networks (GNNs), Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and Multilayer Perceptrons (MLP). Different molecular descriptors (e.g., WHIM, PhysChem, MACCS, ECFP, SMILES) were employed to represent the molecules. The performance was primarily assessed using Root Mean Squared Error (RMSE), with a specific focus on RMSE on activity cliff compounds (RMSEcliff) versus overall RMSE. Principal Component Analysis (PCA) was used for global ranking of methods. The correction involved retraining these models on a corrected dataset after identifying and fixing a bug in the data splitting process.

Datasets

The paper utilizes several publicly available datasets, each corresponding to a specific biological target and response type (e.g., inhibition constant $K_i$ or half-maximal effective concentration $EC_{50}$). These datasets cover a range of targets such as androgen receptor (AR), cannabinoid receptor 1 (CB1), coagulation factor X (FX), and many others. The correction specifically addresses the data splitting for these existing datasets. No new benchmark datasets were introduced in this correction.

Key Insights and Ideas

  • Confirmation of Original Findings: Despite the bug and subsequent retraining, the core conclusions of the original paper regarding the limitations of ML models on activity cliffs remain valid. This reinforces the idea that activity cliffs are indeed a significant challenge for current ML approaches in cheminformatics.
  • Impact of Data Splitting: The correction highlights the critical importance of accurate data splitting, even in seemingly minor aspects like handling activity cliff pairs. A subtle bug can lead to misleading results and impact the perceived performance of models.
  • Activity Cliffs as a True Test: The study consistently shows that models perform worse on activity cliff compounds than on the general dataset, underscoring that these specific molecular variations are where ML models often falter.

Key Results

The correction involved retraining models and updating figures and tables. The paper presents updated Table 1, which details the corrected dataset overview, including the number of molecules and the percentage of activity cliffs. Figures 3, 4, and 5 are also updated to reflect the performance of various traditional and deep learning models on the corrected data. While specific quantitative improvements due to the correction are not the primary focus, the updated figures and tables show the re-evaluated RMSE and RMSEcliff values for different algorithms and descriptors. The general trend of ML models performing worse on activity cliffs compared to overall predictions persists across the updated analyses.

Limitations and Future Work

The paper itself is a correction, so its primary limitation is the initial oversight of the software bug. The authors acknowledge that the same findings and conclusions as the original paper were observed, implying that the fundamental limitations of ML on activity cliffs are robust. The updated code, data, and supporting information are made available on GitHub, facilitating further research and verification. Future work would likely involve developing more robust ML architectures and feature representations that can better capture the subtle structural nuances responsible for activity cliffs, thereby improving generalization and predictive accuracy in these challenging scenarios.

Summary updated

BibTeX
@article{tilborg2024correction,
  author = {Derek van Tilborg and Alisa Alenicheva and Francesca Grisoni},
  title = {{Correction to "Exposing the Limitations of Molecular Machine Learning with Activity Cliffs"}},
  journal = {Journal of Chemical Information and Modeling},
  year = {2024},
  url = {https://openreview.net/forum?id=gGsccgchG0},
}
Figure