TL;DR
This paper introduces PointVS, a new machine learning scoring function for drug discovery that aims to overcome the common problem of models learning data biases rather than true binding physics. By carefully filtering training data and using an interpretable "attention" mechanism, PointVS not only achieves competitive performance in predicting molecular binding but also provides insights into crucial binding interactions, enabling better molecule design.
What Problem Does This Paper Address?
The field of drug discovery heavily relies on computational methods like virtual screening to identify potential drug candidates. A key component of this is "scoring functions," which predict how strongly a small molecule will bind to a target protein. While machine learning (ML) has shown great promise in developing these scoring functions, a significant challenge is their tendency to "memorize" biases present in the training data rather than learning the underlying physical principles of molecular interactions. This leads to models that perform well on data similar to what they've seen before but fail to generalize to new, unseen targets or molecules. This lack of generalizability severely limits their utility in discovering drugs for novel targets.
Methods
The core of this work is the development of PointVS, a novel E(n)-equivariant graph neural network (EGNN). Unlike traditional convolutional neural networks (CNNs) that can lose atomic information as data passes through layers, EGNNs preserve this information, making them more amenable to interpretability. PointVS incorporates an attention mechanism that assigns importance scores to the "edges" (representing atomic interactions) within the graph. This allows the model to learn which interactions are most critical for binding.
A crucial methodological innovation is the rigorous filtering of training and testing datasets. This process aims to remove structurally similar protein-ligand pairs between the training and testing sets to mitigate data leakage and prevent the model from learning spurious correlations.
To understand what PointVS has learned, the authors employ input attribution techniques, specifically edge attention analysis. This allows them to visualize which atomic interactions the model deems most important for a given prediction. This information is then used for fragment elaboration, a drug design strategy where small molecular fragments are grown into more potent lead compounds.
Datasets
The primary datasets used in this study are:
- PDBBind v.2020 General Set: Used for training and finetuning the MLBSF.
- PDBBind Core Set: Used as the primary test set for evaluating binding affinity prediction, replicating the CASF-2016 benchmark.
- Redocked dataset: Used for pre-training the pose classifier.
- gninaSetpose: Used as a test set for pose prediction, allowing comparison with previous work.
The authors also created filtered subsets of these datasets by removing protein-ligand pairs with high sequence identity (proteins) or Tanimoto similarity (ligands) to the test sets. This debiased approach is a key contribution.
Key Insights and Ideas
- Data leakage is a pervasive problem in ML for drug discovery: The study highlights how standard practices in benchmark datasets like CASF-2016 can lead to overly optimistic performance estimates due to unintended overlap between training and testing data.
- Debiasing training data is crucial for generalizability: Rigorous filtering of training data, while slightly impacting performance on some tasks, is essential for ensuring that ML models learn physical interactions rather than dataset biases.
- Interpretable ML models can reveal binding physics: The attention mechanism in PointVS, combined with attribution techniques, allows researchers to directly visualize and understand which atomic interactions the model prioritizes, providing evidence that it's learning genuine binding principles.
- Attribution can guide fragment-based drug design: By extracting "hotspot" information from protein-ligand complexes using PointVS, researchers can more effectively guide the growth of molecular fragments into promising drug candidates, leading to improved docking scores.
Key Results
- Competitive Performance: PointVS, even after rigorous data filtering, achieves performance comparable to leading ML and non-ML scoring functions on the CASF-2016 benchmark for both pose selection and binding affinity prediction.
- Pose Ranking: On the gninaSetpose benchmark, PointVS trained on a debiased set achieved a Top-1 pose ranking of 68%, only slightly lower than the 70% achieved by gnina trained on a biased set.
- Scoring Power: PointVS trained on the debiased PDBBind Core set achieved a Pearson correlation coefficient (PCC) of 0.754, demonstrating its ability to predict binding affinities. This is significantly better than Autodock Vina (0.601) and comparable to other ML methods when trained on biased data.
- Attribution Agreement: PointVS's attribution results showed a strong correlation with the PLIP (Protein-Ligand Interaction Profiler) analysis, indicating that the model correctly identifies important hydrogen bonds and other interactions.
- Fragment Elaboration Improvement: Using PointVS-derived hotspots for fragment elaboration resulted in higher standardized ligand efficiencies (ASLE20) compared to using traditional data-driven hotspot maps, demonstrating the practical utility of the model's insights.
Limitations and Future Work
- Residual Bias: Despite extensive filtering, the authors acknowledge that some degree of data leakage may still exist due to the inherent similarity of many protein-ligand complexes in the PDBBind dataset. This means performance estimates might still be slightly optimistic.
- Computational Cost of Attribution: While edge attention with PointVS is more efficient than other methods, attribution can still be computationally intensive, especially for large datasets.
- Future Work: The authors suggest that their attribution technique could be further applied to less well-studied targets with limited fragment screen data. They also envision broader applications of debiased ML models and attribution techniques for molecule generation and design.
Summary updated