MAIP: a web service for predicting blood‐stage malaria inhibitors

Nicolas Bosc, Eloy Félix, Ricardo Arcila, David Méndez, Martin Saunders, Darren V. S. Green, Jason Ochoada, Anang A. Shelat, Éric Martin, Preeti Iyer, Ola Engkvist, Andreas Verras, James Duffy, Jeremy N. Burrows, Mark Gardner, Andrew R. Leach

TL;DR

This paper introduces MAIP, a freely accessible web service that predicts potential blood-stage malaria inhibitors using a consensus machine learning model. By leveraging an open-source platform and combining data from multiple partners, MAIP aims to accelerate antimalarial drug discovery, addressing the critical need for new treatments against drug-resistant malaria strains.

What Problem Does This Paper Address?

Malaria remains a devastating global health challenge, particularly in developing countries, causing hundreds of thousands of deaths annually. The emergence of drug resistance to current treatments, like artemisinin-based combination therapies (ACTs), necessitates the urgent discovery of new, effective antimalarial drugs. While high-throughput screening has advanced, building large, comprehensive datasets for machine learning models is hampered by the proprietary nature of much bioactivity data. This makes it difficult to integrate diverse datasets and develop robust predictive models. Previous attempts to create consensus models faced limitations due to proprietary software and a lack of a publicly available prediction platform.

Methods

The researchers built upon a prior study that developed a consensus machine learning model for predicting antimalarial activity. This new work focuses on using open-source tools and a larger, combined dataset. Key methodological aspects include:

  • Open-Source Software Development: They developed a platform using Python and the RDKit cheminformatics toolkit for molecule standardization, descriptor calculation, and model building. This ensures reproducibility and allows for community contributions.
  • Molecule Standardization: A critical step to ensure consistency across diverse datasets, using an open-source RDKit-based standardizer to handle tautomeric forms and identify parent molecules.
  • Descriptor Calculation: Various combinations of molecular descriptors, including Extended Connectivity Fingerprints (ECFP), Functional-Class Fingerprints (FCFP), and physico-chemical properties (logP, molecular weight, etc.), were explored.
  • Model Building: A Naïve Bayes classifier was employed, consistent with the previous study, to predict the probability of a compound inhibiting blood-stage malaria.
  • Consensus Approaches: Several methods were evaluated to combine individual models into a single, more robust predictor. These included MaxScore (taking the maximum prediction score from any model), MeanScore (averaging scores), MinRank (minimum rank), and a "metamodel" approach that merges all partner models into a single, larger Naïve Bayes model.
  • Reproducibility Assessment: The study explicitly aimed to replicate previous findings using open-source tools, investigating the impact of different fingerprint implementations on model performance.

Datasets

The study utilized a combination of previously described and newly added datasets for training and validation:

  • Training Datasets: Eleven datasets from five different partners were used. These included datasets from Evotec, Johns Hopkins, MRCT, MMV-St. Jude, AZ, GSK, and St. Jude Vendor Library, largely mirroring those from the previous study. Additionally, three new datasets from the Medicines for Malaria Venture (MMV5, MMV6, MMV7) and a significantly modified Novartis dataset were incorporated. These datasets are proprietary to the respective partners.
  • Validation Datasets: Three datasets were used for external validation: the MMV test set (from the previous study), the PubChem dataset, and the St. Jude Screening Set. These were used to assess the performance and generalizability of the developed models.

Key Insights and Ideas

  • The Power of Open Source: The shift to open-source tools (RDKit, Python) not only enabled reproducibility but also democratized the process, allowing for broader community participation and model extension.
  • Consensus Models Outperform Individuals: Combining multiple models, even those trained on proprietary data, generally leads to more robust and accurate predictions than any single model alone.
  • Metamodel Efficiency: The "metamodel" approach, which consolidates all partner models into a single, larger model, offers significant computational advantages for real-time predictions without sacrificing performance.
  • Reproducibility Challenges: Even with open-source tools, minor differences in descriptor calculation (e.g., ECFP vs. FCFP implementations) can lead to variations in model performance, highlighting the importance of precise methodological descriptions.
  • Data Diversity Matters: The performance of models varied across validation sets, suggesting that the chemical space covered by each training dataset influences its predictive power on different types of compounds.

Key Results

  • Robust Model Performance: The developed consensus models, particularly the metamodel, demonstrated good performance across multiple validation sets, achieving ROC AUC scores generally above 0.65 and showing significant enrichment factors (EF[1%] and EF[10%]) for active compounds.
  • Metamodel Efficiency: The metamodel approach achieved performance comparable to other consensus methods (like MaxScore) but with substantial computational advantages due to dealing with a single model.
  • Reproducibility Achieved: The study successfully reproduced the performance of the previous consensus model using open-source tools, with ECFP implementations showing near-perfect identity and FCFP implementations showing good correlation.
  • MAIP Web Service: A user-friendly web application, MAIP, was developed and made publicly available, allowing researchers to upload molecules and receive predictions for their antimalarial potential.

Limitations and Future Work

  • Proprietary Training Data: While the models are public, the underlying training data remains proprietary, limiting direct analysis of the training set's chemical space and potential biases.
  • Domain of Applicability: Directly assessing the domain of applicability for the consensus models was challenging due to the obscured nature of the training data.
  • Further Validation: The authors encourage users to employ additional in silico filters and experimental validation, as high-scoring compounds might possess undesirable physicochemical properties or known antimalarial activity.
  • Future Contributions: The open-source nature of the platform is intended to facilitate contributions from new partners and the inclusion of additional malaria bioassay data, potentially expanding the model's scope and accuracy. The approach is also adaptable to other drug discovery areas.

Summary updated

Abstract

Malaria is a disease affecting hundreds of millions of people across the world, mainly in developing countries and especially in sub-Saharan Africa. It is the cause of hundreds of thousands of deaths each year and there is an ever-present need to identify and develop effective new therapies to tackle the disease and overcome increasing drug resistance. Here, we extend a previous study in which a number of partners collaborated to develop a consensus in silico model that can be used to identify novel molecules that may have antimalarial properties. The performance of machine learning methods generally improves with the number of data points available for training. One practical challenge in building large training sets is that the data are often proprietary and cannot be straightforwardly integrated. Here, this was addressed by sharing QSAR models, each built on a private data set. We describe the development of an open-source software platform for creating such models, a comprehensive evaluation of methods to create a single consensus model and a web platform called MAIP available at https://www.ebi.ac.uk/chembl/maip/ . MAIP is freely available for the wider community to make large-scale predictions of potential malaria inhibiting compounds. This project also highlights some of the practical challenges in reproducing published computational methods and the opportunities that open-source software can offer to the community.
BibTeX
@article{bosc2021maip,
  author = {Nicolas Bosc and Eloy Félix and Ricardo Arcila and David Méndez and Martin Saunders and Darren V. S. Green and Jason Ochoada and Anang A. Shelat and Éric Martin and Preeti Iyer and Ola Engkvist and Andreas Verras and James Duffy and Jeremy N. Burrows and Mark Gardner and Andrew R. Leach},
  title = {{MAIP: a web service for predicting blood‐stage malaria inhibitors}},
  journal = {Journal of Cheminformatics},
  year = {2021},
  doi = {10.1186/s13321-021-00487-2},
}
Figure