TL;DR
This paper tackles the critical issue of bias in compound-protein interaction (CPI) prediction datasets, which hinders model generalization. They introduce CDPN, a novel protocol for creating a debiased benchmark dataset, and use it to systematically evaluate existing deep learning models, identifying top performers and revealing limitations in current approaches.
What Problem Does This Paper Address?
Accurately predicting how small molecules (compounds) interact with proteins is fundamental to drug discovery. It helps identify potential drug candidates, refine existing ones, and predict off-target effects. While deep learning (DL) models have shown promise, they often learn "shortcuts" due to biases in existing datasets. These biases include over-represented molecular scaffolds and imbalanced distributions of positive and negative interactions. This leads to models that perform well on familiar data but fail to generalize to new, unseen compounds or proteins, making them unreliable for real-world drug discovery. Existing debiasing methods often compromise data diversity, creating a trade-off that this paper aims to overcome.
Methods
The core of this work is the CDPN (Clustering-based Down-sampling and Putative Negatives) protocol. It's a three-phase approach:
- Chemical Space Clustering: They start with a massive dataset of ~114 million PubChem compounds, down-sample it to ~9.5 million, and then cluster them into ~75,745 structurally distinct groups using k-means clustering on PubChem fingerprints. This ensures representation of diverse chemical scaffolds, including rare ones.
- Cluster-Aware Down-sampling: To mitigate analogue bias (where similar compounds are over-represented), they retain a maximum of 3 interactions per cluster-label combination for each target. This prevents specific scaffolds from dominating the training data.
- Putative Negative Generation: To balance label distributions, they generate "putative negatives." For targets with an excess of positive interactions, negatives are sampled from clusters devoid of known interactions. For compounds with only positive annotations, negatives are generated from protein families outside their known interaction partners. This aims to create a more balanced and representative dataset.
Beyond CDPN, the paper also involves:
- Systematic Benchmarking: They evaluate over 40 existing sequence-based CPI models, focusing on different components like compound encoders, protein encoders (especially protein language models - PLMs), and interaction decoders.
- Ablation Studies: They dissect the contributions of individual model components to understand their impact on generalization.
- Interpretability Analysis: They assess the interpretability of attention mechanisms in CPI models using PDBbind structures.
- DeepSEQreen Web Server: The top-performing models are integrated into a user-friendly web server for broader accessibility.
Datasets
- PubChem: Used as the source for the initial ~114 million compounds for clustering.
- ChEMBL v33, BindingDB v202312, PDBbind v2020: These established databases were used for data cleaning, integration, and forming the initial "Full" dataset.
- CDPN Dataset: The newly constructed debiased dataset, comprising 3879 protein targets, 289,287 unique compounds, and 741,806 interactions. This is the primary benchmark dataset introduced by the paper.
- Enamine Library: Used for virtual screening evaluations.
- DUD-E: Another dataset used for virtual screening evaluation.
Key Insights and Ideas
- Bias is a Major Hurdle: Existing CPI datasets are heavily biased, leading models to exploit shortcuts (like positive-to-negative ratios) rather than learning genuine interaction mechanisms. This significantly hinders generalization.
- CDPN Balances Diversity and Representation: The CDPN protocol successfully mitigates analogue bias and imbalanced labels while preserving chemical space diversity, creating a more robust benchmark.
- Protein Language Models (PLMs) are Promising, but Nuances Exist: While PLMs generally enhance generalization, the specific PLM and how it's integrated (e.g., embeddings vs. fine-tuning) matters. Simpler integration methods like embeddings can be competitive, and parameter size isn't always the sole determinant of performance.
- Attention Mechanisms Lack Interpretability: Despite being designed for interpretability, attention modules often assign high weights to non-pocket regions, suggesting they don't reliably highlight key binding sites.
- Compound Encoders Matter for Generalization: Pretrained encoders like KPGT and Mol2Vec, along with graph-based methods like SAGE, show better generalization, especially in unseen scenarios, compared to simpler fingerprint-based methods.
Key Results
- CDPN Improves Prediction Reliability: Visualizations show that before CDPN, models assigned disproportionately high probabilities to majority compounds/targets. After CDPN, score distributions are more calibrated, indicating reduced bias.
- Enhanced Virtual Screening: Models trained on the CDPN dataset consistently outperformed those trained on pre-CDPN data across various virtual screening metrics (BEDROC, EF1%) for both seen and unseen targets.
- Top Performing Models Identified: KPGT-Ankh and Mol2Vec-ESM2-8M combinations demonstrated superior generalization and virtual screening performance, especially in unseen scenarios.
- Attention Mechanism Limitations: The study found that attention mechanisms did not yield significant performance improvements and had low interaction-over-union (IoU) scores when evaluated for interpretability against PDBbind structures.
- DeepSEQreen Launch: A user-friendly web server integrating top models is now available, facilitating drug discovery research and community feedback.
Limitations and Future Work
- Negative Prevalence: CDPN cannot fully address targets or compound clusters with extremely high negative prevalence, as assuming putative positives without evidence is not feasible. This remains a challenge for modeling.
- CPA Prediction: While CDPN shows modest improvements for compound-protein affinity (CPA) prediction, regression tasks are inherently more challenging, and further work is needed for regression-balanced datasets.
- Interpretability of Attention: The lack of reliable interpretability in attention mechanisms suggests future work should explore integrating structure-aligned PLMs or incorporating known binding pocket information as supervision.
- Fine-tuning PLMs: The paper notes that fine-tuning PLMs within CPI models can improve generalization, suggesting this as a key direction for future research.
- Structure-Aware Models: The authors suggest that future protein language models that align sequence with structure during pretraining could offer more structure-aware representations for CPI tasks.
Summary updated