TL;DR
This paper investigates the robustness and generalization capabilities of deep learning models designed to predict drug-drug interactions (DDIs) based on molecular structures. The authors find that while these models perform well on known drug pairs, they struggle to generalize to new, unseen drugs, highlighting a critical limitation for real-world applications. They propose that more rigorous evaluation schemes and data augmentation techniques are crucial for developing truly reliable DDI prediction models.
What Problem Does This Paper Address?
Predicting drug-drug interactions (DDIs) is crucial for patient safety, as drug combinations can lead to unexpected and potentially harmful effects. Traditional methods for identifying DDIs, like in vivo experiments and clinical trials, are time-consuming, expensive, and often fail to detect interactions until after a drug is approved. In silico methods, particularly those using deep learning and molecular structures, offer a faster and more cost-effective alternative. However, a significant challenge remains: the ability of these models to generalize their predictions to drugs they haven't encountered during training. This lack of generalization is a major hurdle for their widespread adoption in drug discovery and clinical practice, as it limits their utility in predicting interactions involving novel drug candidates. The paper aims to rigorously evaluate the generalization performance of these structure-based DDI models and identify factors that contribute to their limitations.
Methods
The study employs a general deep learning architecture that encapsulates various recent structure-based DDI models. This architecture involves:
- Feature Extraction: Using different molecular representations to extract features from individual drugs. The paper explores three main types:
- ECFP6 fingerprints: Binary vectors representing the chemical environment around atoms.
- SMILES (Simplified Molecular-Input Line-Entry System): String-based representations of chemical compounds.
- Molecular Graphs: Representing molecules as nodes (atoms) and edges (bonds).
- Model Architectures: For each representation, specific deep learning models are used:
- DeepDDI: Uses ECFP6 fingerprints with a feedforward neural network.
- 1D-CNN and BLSTM: Process SMILES strings.
- GIN and GIN + LaPool: Utilize molecular graphs with Graph Neural Networks.
- Multilabel Classification: The models are framed as multilabel classifiers, predicting multiple potential phenotypes (side effects or PK/PD effects) for a given drug pair.
- Evaluation Schemes: The core of the study lies in evaluating models under three distinct splitting strategies to assess generalization:
- Random Split: Randomly partitions drug pairs into training, validation, and testing sets. This is suitable for discovering DDIs within a known set of drugs.
- One-Unseen Split: Ensures that for each pair in the validation/testing set, at least one drug was not seen during training. This simulates scenarios where new drugs are combined with existing ones.
- Both-Unseen Split: Guarantees that both drugs in a pair are entirely new to the training set. This is the most challenging scenario, simulating the prediction of interactions between two novel drug candidates.
- Data Augmentation: Explored as a technique to improve generalization by generating multiple SMILES variations for each molecule.
- Multitask Learning: Investigated by adding an auxiliary task (predicting Connectivity Map scores) to learn shared drug representations.
Datasets
The study utilizes three publicly available datasets:
- DrugBank: A repository of experimentally discovered PK/PD interactions and clinically validated side effects.
- Twosides: A database mined from text sources, containing reported unsafe co-drug prescriptions.
- Twosides-NOSYN: A smoothed version of Twosides, where semantically similar side effects are grouped using the Unified Medical Language System (UMLS) Metathesaurus.
Key Insights and Ideas
- Generalization is a Major Bottleneck: The most striking finding is that DDI prediction models, despite achieving high performance on known drug pairs (random split), exhibit a significant drop in performance when evaluated on unseen drugs (one-unseen and both-unseen splits). This suggests that current models are better at memorizing patterns within known drugs rather than truly understanding the underlying molecular mechanisms of DDIs.
- Evaluation Scheme Matters Critically: The choice of data splitting strategy has a profound impact on reported model performance. Overly optimistic results from random splits can mask the true limitations of models when faced with novel drugs. The paper strongly advocates for using more challenging evaluation schemes like one-unseen and both-unseen for a realistic assessment.
- Molecular Representation Influences Performance: Different molecular representations (SMILES, fingerprints, graphs) and their associated feature extractors lead to varying performance levels. Graph-based models (GIN) showed promise in the most challenging "both-unseen" scenario, suggesting their potential for better generalization.
- Imbalance is a Persistent Challenge: Both intra-class (positive vs. negative samples for a phenotype) and inter-class (uneven distribution of phenotypes) imbalances in DDI datasets significantly impact model performance and generalization. Low-frequency phenotypes are particularly affected by the evaluation scheme's difficulty.
- Data Augmentation Offers Some Relief: While not a complete solution, data augmentation (generating multiple SMILES variants) was shown to reduce the performance gap between different evaluation schemes, indicating its potential to improve robustness.
- Multitask Learning Falls Short (in this context): Adding an auxiliary task (CMAP score prediction) did not significantly improve generalization for DDI prediction, suggesting that the chosen auxiliary task might not share the optimal inductive bias or that domain adaptation might be a more suitable approach.
Key Results
- Performance Drop: Across all tested models and datasets, performance metrics (AUROC and AUPRC) consistently decreased from random split to one-unseen to both-unseen evaluation schemes. For instance, models performed significantly better on DrugBank under random split than under the both-unseen split.
- Model Stability: The models generally exhibited good stability, with small standard deviations in performance across multiple runs, indicating that their performance is not highly sensitive to random variations in training data.
- SMILES vs. Graphs: SMILES-based models (1D-CNN, BLSTM) generally outperformed fingerprint and graph-based models in random and one-unseen splits. However, graph models (GIN) showed better performance in the more stringent both-unseen split.
- Data Augmentation Impact: Using augmented SMILES data reduced the performance gap between evaluation schemes, with optimal results achieved at specific augmentation levels (e.g., n=20 for Twosides, n=80 for DrugBank).
- False Negatives and Novel DDIs: The study identified potential false negatives (interactions labeled as negative but predicted as positive by the model) and uncovered potential new DDIs with supporting evidence from literature, demonstrating the models' ability to identify previously unknown interactions, albeit with varying confidence depending on the evaluation scheme.
Limitations and Future Work
- Lack of Standardization: The paper highlights the absence of standardized training and evaluation protocols as a major impediment to comparing DDI models and assessing their true capabilities.
- Imbalanced Data: The inherent imbalance in DDI datasets poses a significant challenge for model learning and generalization. The authors suggest the need for more comprehensive datasets or advanced techniques for handling imbalance in multilabel classification.
- Domain Adaptation: The authors suggest that domain adaptation (or transfer learning) could be a more promising avenue for improving the prediction of side effects for new drugs, given that multitask learning with the chosen auxiliary task did not yield significant improvements.
- Experimental Validation: The paper emphasizes the need for experimental validation of interactions reported in data-mined databases like Twosides, suggesting that curated reference sets are preferable.
- Future Directions: The authors recommend using disjointed evaluation schemes (one-unseen, both-unseen) for training models intended for predicting interactions with new drugs. They also propose exploring domain adaptation techniques and developing more robust feature extractors that are less sensitive to input variations.
Summary updated