Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings

Remzi Celebi, Hüseyin Uyar, Erkan Yasar, Özgür Gümüs, Oguz Dikenelli, Michel DumontierBMC Bioinformatics 2019 · DOI

TL;DR

This paper tackles the challenge of predicting drug-drug interactions (DDIs) by employing knowledge graph embeddings. They introduce a more realistic evaluation framework, "disjoint cross-validation," to better assess model performance on drugs with limited or no known interactions, a crucial step for drug discovery and safety.

What Problem Does This Paper Address?

Adverse Drug Events (ADEs) are a significant public health concern, with drug-drug interactions (DDIs) being a major contributor. While existing methods for identifying DDIs are valuable, they often suffer from systematic biases in their evaluation, leading to overly optimistic performance metrics. This is particularly problematic when predicting interactions for new drugs that have little to no prior interaction data (the "cold-start" problem). Traditional cross-validation can inadvertently include information about drug pairs in both the training and testing sets, masking the true predictive power of models for novel interactions. This paper aims to bridge this gap by proposing and evaluating DDI prediction methods in more realistic, "cold-start" scenarios.

Methods

The core of this research lies in leveraging knowledge graph embeddings to represent drugs as dense vectors. These embeddings are learned from large biological knowledge graphs. The authors explore several established graph embedding techniques, including:

  • RDF2Vec: An adaptation of word2vec for RDF graphs, which learns vector representations of entities by treating graph walks as sequences. They specifically tested both the Continuous Bag-of-Words (CBOW) and Skip-Gram (SG) architectures.
  • TransE: A translation-based embedding model that represents entities and relations as vectors, aiming to satisfy the relation h + r ≈ t for triples (h, r, t).
  • TransD: An extension of TransE that uses projection vectors to handle more complex relationships (one-to-many, many-to-one, many-to-many).

After generating these feature vectors for drugs, they are fed into standard machine learning classifiers: Logistic Regression, Naive Bayes, and Random Forest, to predict DDIs.

The key methodological innovation is the introduction of disjoint cross-validation (CV) schemes. Instead of traditional CV, they propose:

  • Drug-wise disjoint CV: Where drugs in the test set are entirely absent from the training set. This simulates predicting interactions for drugs with no known interactions.
  • Pairwise disjoint CV: Where both drugs in a test pair are absent from the training set, representing a more stringent cold-start scenario.

Datasets

The study utilizes several well-established biological knowledge graph datasets:

  • DrugBank v4 and v5: Comprehensive online databases containing drug information, including drug structure, pathways, and interaction data. DrugBank v5, the later version, was primarily used for evaluation due to its larger size and more extensive data.
  • KEGG: A collection of databases for biological pathways and systems, from which DDIs were extracted.
  • PharmGKB: A resource for pharmacogenomic information, also used to extract DDI data.

The authors explicitly state that DDI links were removed from these knowledge graphs before embedding to avoid bias in the prediction task.

Key Insights and Ideas

  • Realistic Evaluation is Crucial: The paper strongly emphasizes that traditional CV methods for DDI prediction are overly optimistic and do not reflect real-world scenarios, especially for new drugs. Disjoint CV is presented as a more honest assessment.
  • Knowledge Graph Embeddings are Powerful: The study demonstrates that learned drug embeddings from knowledge graphs can achieve performance comparable to existing state-of-the-art methods, even in challenging cold-start settings.
  • RDF2Vec with Skip-Gram Shines: Among the tested embedding methods, RDF2Vec, particularly with the Skip-Gram architecture, consistently performed best, suggesting its effectiveness in capturing complex relationships within biological knowledge graphs.
  • Single Knowledge Graph Sufficiency: Surprisingly, integrating multiple knowledge graphs (DrugBank, KEGG, PharmGKB) did not significantly improve prediction performance compared to using DrugBank alone. This suggests that a comprehensive single source like DrugBank might be sufficient for this task.

Key Results

  • Performance Drop in Disjoint Settings: As expected, performance metrics (AUC, AUPR, F1-score) dropped significantly when moving from traditional CV to drug-wise and pairwise disjoint CV. This highlights the bias in traditional methods.
  • RDF2Vec (SG) with Random Forest Achieves Top Scores: For traditional CV, RDF2Vec with Skip-Gram and Random Forest achieved the best scores, with an AUC of 0.93, AUPR of 0.92, and F1-score of 0.85 on the DrugBank dataset.
  • Disjoint CV Performance: Even in the more challenging disjoint settings, the models, particularly Random Forest, showed respectable performance, indicating their utility for predicting novel DDIs. For instance, in pairwise disjoint CV, Random Forest achieved an F1-score of 0.75.
  • Logistic Regression and Naive Bayes Show Promise: While Random Forest was often the best, Logistic Regression and Naive Bayes showed competitive or better performance in some disjoint and time-slice settings, suggesting their robustness.

Limitations and Future Work

  • Lack of Mechanistic Explanations: A key limitation of knowledge graph embedding methods, as acknowledged by the authors, is their "black box" nature. They provide predictive power but do not inherently offer mechanistic explanations for the predicted DDIs.
  • Future Work - Interpretability: The authors suggest that future work could focus on mining rules or paths within knowledge graphs to provide more interpretable outcomes for drug pairs.
  • Future Work - Negative Sampling: The study acknowledges that negative sampling strategies were not the primary focus but could be an area for further exploration.
  • Future Work - Temporal Evaluation: While a time-slice CV was explored, its applicability is limited by the availability of multiple data snapshots. Further research into temporal evaluation methods could be beneficial.
  • Future Work - Network Theoretic Perspective: The authors suggest that a deeper analysis from a network theoretic perspective (e.g., scale-freeness) could lead to a more comprehensive assessment of DDI prediction methods.

Summary updated

BibTeX
@article{celebi2019evaluation,
  author = {Remzi Celebi and Hüseyin Uyar and Erkan Yasar and Özgür Gümüs and Oguz Dikenelli and Michel Dumontier},
  title = {{Evaluation of knowledge graph embedding approaches for drug-drug interaction prediction in realistic settings}},
  journal = {BMC Bioinformatics},
  year = {2019},
  doi = {10.1186/s12859-019-3284-5},
}
Figure