Target prediction utilising negative bioactivity data covering large chemical space

Lewis Mervin, Avid M. Afzal, Georgios Drakakis, Richard P. Lewis, Ola Engkvist, Andreas BenderJournal of Cheminformatics 2015 · DOI

TL;DR

This paper introduces a novel in silico target prediction method that leverages both active and inactive bioactivity data to improve the accuracy and early recognition capabilities of predicting drug targets. By incorporating a vast amount of negative data, their approach demonstrates superior performance compared to models trained solely on active compounds, offering a more comprehensive understanding of chemical space for drug discovery.

What Problem Does This Paper Address?

Identifying the specific biological targets of small molecules, known as "target deconvolution," is a critical but challenging step in drug discovery. Phenotypic screening often yields compounds with unknown mechanisms of action (MOA). While computational methods exist to predict these targets, many rely solely on active bioactivity data, neglecting the vast repositories of inactive data. This omission can lead to incomplete models that don't fully capture the nuances of chemical space, potentially hindering accurate target prediction, especially for orphan compounds. This work aims to bridge this gap by integrating large-scale negative bioactivity data into target prediction models.

Methods

The core of this research is a Bernoulli Naïve Bayes classifier implemented using the Scikit-learn library. This algorithm was chosen for its efficiency, ability to handle imbalanced datasets, and its capacity to interpret binary bit string features from compound fingerprints.

Key methodological innovations include:

  • Construction of a novel human bioactivity dataset: This involved assimilating over 195 million bioactivity data points from the ChEMBL and PubChem repositories.
  • Sphere Exclusion (SE) algorithm for oversampling presumed inactive compounds: To address the scarcity of experimentally confirmed inactive compounds for many targets, the SE algorithm was used to sample presumed inactive compounds that were sufficiently dissimilar to known active compounds. This helps to create more balanced training sets.
  • Target-specific activity threshold generation: To mitigate the issue of class imbalance, custom thresholds were developed for each target. These thresholds are determined by evaluating the performance (precision, recall, accuracy, F1-score) across a range of probability cut-offs.
  • Distance-based applicability domain (AD) analysis: This method assesses the reliability of predictions by measuring the similarity between query compounds and the training set.

Datasets

  • ChEMBL (Version 18): Used to construct the active bioactivity training dataset. Bioactivities with IC50/EC50/K₁/K₁ values of 10 μM or lower and a CONFIDENCE_SCORE of 5 or greater were extracted.
  • PubChem Compound and PubChem BioAssay databases: Mined for inactive compounds. An automated extraction pipeline was developed to obtain inactive compound-ligand pairs.
  • WOMBAT database: Used as an external validation dataset to assess the models' performance in a real-world setting.

Key Insights and Ideas

  • Negative data is crucial: The inclusion of inactive bioactivity data significantly improves target prediction models, leading to superior AUC and better early recognition capabilities. This suggests that understanding what a compound doesn't do is as important as understanding what it does do.
  • Intraclass similarity and class size matter: Model performance is heavily influenced by the similarity within the active training set and the number of compounds in a class. Larger classes with higher internal similarity generally lead to better predictions.
  • Sphere exclusion is effective for balancing data: The SE algorithm successfully addresses the issue of insufficient inactive data by sampling dissimilar inactive compounds, leading to better performance compared to using only PubChem inactive classes.
  • Applicability domain is key for reliability: A distance-based AD analysis reveals that predictions are more reliable when query compounds are similar to the training set. A Tanimoto coefficient (Tc) cut-off of around 0.3 can be used to indicate confidence in predictions.
  • Threshold selection depends on the application: Different decision thresholds (precision, recall, F1-score, accuracy) offer trade-offs between precision and recall, allowing users to tailor the model's output to specific needs (e.g., safety profiling vs. hit identification).

Key Results

  • Internal Validation: The models achieved a mean recall of 67.7% and precision of 63.8% for active compounds, and an impressive 99.6% recall and 99.7% precision for inactive compounds.
  • External Validation (WOMBAT): The models demonstrated an average precision-recall AUC of 0.56 and a BEDROC score of 0.85 when trained with both active and inactive data. This is a significant improvement over models trained solely on active data, which achieved an AUC of 0.45 and a BEDROC of 0.76.
  • Precision Thresholds: The precision threshold was found to be the most enriched metric, balancing precision and recall effectively.
  • Sphere Exclusion Impact: For targets with over 1000 confirmed inactives, SE sampling improved WOMBAT prediction performance in 19 out of 20 cases.

Limitations and Future Work

  • WOMBAT data assumptions: The external validation using WOMBAT data relies on the assumption that unannotated compounds are inactive. This could lead to false positives if a compound is actually active but not yet annotated.
  • Class imbalance challenges: While addressed, class imbalance remains a challenge, particularly for smaller classes or targets with very few active compounds.
  • Future work: The authors have made their realized target prediction protocol, PIDGIN, available on GitHub, encouraging further development and application of their method. The study highlights the importance of continued exploration of confidence scores and their impact on data set extraction.

Summary updated

Abstract

The inclusion of inactive data for model training produces models with superior AUC and improved early recognition capabilities, although the results from internal and external validation of the models show differing performance between the breadth of models. The realised target prediction protocol is available at https://github.com/lhm30/PIDGIN.Graphical abstractThe inclusion of large scale negative training data for in silico target prediction improves the precision and recall AUC and BEDROC scores for target models.
BibTeX
@article{mervin2015target,
  author = {Lewis Mervin and Avid M. Afzal and Georgios Drakakis and Richard P. Lewis and Ola Engkvist and Andreas Bender},
  title = {{Target prediction utilising negative bioactivity data covering large chemical space}},
  journal = {Journal of Cheminformatics},
  year = {2015},
  doi = {10.1186/s13321-015-0098-y},
}
Figure