PharmaBench: Enhancing ADMET benchmarks with large language models

Zhangming Niu, Xianglu Xiao, Wenfan Wu, Qiwei Cai, Yinghui Jiang, Wangzhen Jin, Minhao Wang, Guojian Yang, Lingkang Kong, Xurui Jin, Guang Yang, Hongming ChenScientific 2024 · DOI

TL;DR

PharmaBench is a new, large-scale open-source benchmark for predicting ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) properties, comprising over 52,000 entries across 11 datasets. The researchers leveraged a multi-agent Large Language Model (LLM) system to automatically extract and standardize experimental conditions from thousands of unstructured bioassay descriptions, significantly improving data quality and model reliability compared to previous benchmarks.

What Problem Does This Paper Address?

In drug discovery, predicting a molecule's ADMET properties is a "make or break" step. If a drug is toxic or doesn't reach its target in the body, it fails, regardless of how well it works in a petri dish. While AI models can help predict these properties, they are only as good as the data they are trained on.

Existing ADMET benchmarks suffer from two major flaws:

  1. Small Scale: Most datasets are too small to train robust deep-learning models.
  2. Context Blindness: Experimental results (like solubility) vary wildly depending on conditions like pH or temperature. In public databases like ChEMBL, these conditions are often buried in unstructured text descriptions rather than neat table columns. Merging data from different labs without accounting for these conditions creates "noisy" data that confuses AI models.

Methods

The authors developed a multi-agent data mining system powered by GPT-4 to turn unstructured assay descriptions into structured metadata. The system uses three specialized agents:

  • Keyword Extraction Agent (KEA): Analyzes a subset of assays to identify the most important experimental conditions (e.g., "pH level," "incubation time").
  • Example Forming Agent (EFA): Takes those keywords and creates "few-shot" learning examples to teach the next agent exactly what to look for.
  • Data Mining Agent (DMA): The workhorse that processes all 14,401 bioassays to extract the specific values for the identified conditions.

Following this LLM-driven extraction, the team applied a rigorous Data Processing Workflow involving structure standardization (using RDKit), filtering for drug-likeness, and removing outliers. Finally, they used "scaffold splitting"—a method that ensures the test set contains molecular structures fundamentally different from the training set—to better simulate real-world drug discovery challenges.

Datasets

The primary source was the ChEMBL database, supplemented by other public sets like AstraZeneca’s LogD and the B3DB (Blood-Brain Barrier) dataset.

The resulting PharmaBench includes 11 datasets covering:

  • Physicochemical: LogD, Water Solubility.
  • Absorption/Distribution: Blood-Brain Barrier (BBB), Plasma Protein Binding (PPB).
  • Metabolism: Three CYP450 isoforms (2C9, 2D6, 3A4).
  • Clearance: Liver Microsome Clearance (Human, Rat, Mouse).
  • Toxicity: AMES mutagenicity.

Key Insights and Ideas

  • LLMs as Expert Curators: The most significant "aha" moment is that LLMs can act as domain-expert data cleaners. By identifying that a solubility test was done at pH 7.4 versus pH 5.0, the system can group "apples with apples," reducing the inherent noise in biological data.
  • Quality Over Quantity: The paper demonstrates that "cleaning" the data using LLM-extracted conditions significantly improved the correlation ($R$) between repeated experiments. For example, in LogD experiments, the correlation coefficient jumped from 0.774 to 0.881 after processing.
  • The "Scaffold" Challenge: The results highlight that models perform much worse on "scaffold splits" than "random splits." This confirms that current AI still struggles to generalize to entirely new classes of molecules, making PharmaBench a vital "stress test" for new architectures.

Key Results

The researchers validated PharmaBench using nine different AI models, ranging from traditional Machine Learning (XGBoost, Random Forest) to advanced Graph Neural Networks and Transformers (Uni-Mol, KANO).

  • Performance Gains: Deep learning models generally outperformed traditional ML in regression tasks (predicting continuous values). For LogD, the best model (Uni-Mol) achieved an $R$ of 0.901.
  • Classification: For binary tasks like the AMES toxicity test, XGBoost remained highly competitive, sometimes surpassing deep learning models, suggesting that "simpler" models are still robust for certain ADMET tasks.
  • Data Consistency: The "Repeated Test" validation showed that the workflow successfully eliminated contradictory entries, resulting in a benchmark where the data points are much more reliable than the raw source material.

Limitations and Future Work

While the LLM system was highly effective for physicochemical properties, the authors noted that CYP metabolism and clearance experiments still show relatively low consistency. This is likely due to the extreme complexity of in vitro biological assays, which may involve variables that even an LLM can't fully capture from a short text description.

Future work involves exploring explainable AI (XAI) to understand why certain molecular features lead to poor ADMET profiles and further refining the data mining process for even more complex biological endpoints.

Summary updated

Abstract

Accurately predicting ADMET (Absorption, Distribution, Metabolism, Excretion, and Toxicity) properties early in drug development is essential for selecting compounds with optimal pharmacokinetics and minimal toxicity. Existing ADMET-related benchmark sets are limited in utility due to their small dataset sizes and the lack of representation of compounds used in drug discovery projects. These shortcomings hinder their application in model building for drug discovery. To address this issue, we propose a multi-agent data mining system based on Large Language Models that effectively identifies experimental conditions within 14,401 bioassays. This approach facilitates merging entries from different sources, culminating in the creation of PharmaBench. Additionally, we have developed a data processing workflow to integrate data from various sources, resulting in 156,618 raw entries. Through this workflow, we constructed PharmaBench, a comprehensive benchmark set for ADMET properties, which comprises eleven ADMET datasets and 52,482 entries. This benchmark set is designed to serve as an open-source dataset for the development of AI models relevant to drug discovery projects.
BibTeX
@inproceedings{niu2024pharmabench,
  author = {Zhangming Niu and Xianglu Xiao and Wenfan Wu and Qiwei Cai and Yinghui Jiang and Wangzhen Jin and Minhao Wang and Guojian Yang and Lingkang Kong and Xurui Jin and Guang Yang and Hongming Chen},
  title = {{PharmaBench: Enhancing ADMET benchmarks with large language models}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41597-024-03793-0},
}
Figure