PanGu Drug Model: Learn a Molecule Like a Human

Xinyuan Lin, Chi Xu, Zhaoping Xiong, Xinfeng Zhang, Ningxi Ni, Bolin Ni, Jianlong Chang, Ruiqing Pan, Zidong Wang, Fan Yu, Qi Tian, Hualiang Jiang, Mingyue Zheng, Nan QiaobioRxiv 2022 · DOI

TL;DR

This paper introduces PanGu Drug Model, a novel deep learning architecture that learns molecular representations from both structural graphs and chemical formulas. Pre-trained on a massive dataset of 1.7 billion molecules, PanGu achieves state-of-the-art results across 20 drug discovery tasks, including property prediction, molecule generation, and optimization, significantly accelerating the drug discovery pipeline.

What Problem Does This Paper Address?

The traditional drug discovery process is notoriously long, expensive, and inefficient, often taking over a decade and hundreds of millions of dollars to bring a single drug to market. While AI has shown promise in accelerating specific steps, a unified model capable of handling the entire drug discovery cycle—from design and screening to optimization—has been lacking. Existing AI models often focus on single tasks or struggle with the complexity and diversity of chemical data. Furthermore, the vast majority of chemical structures lack detailed annotations, making it challenging to train robust models. This paper aims to bridge this gap by developing a powerful, versatile pre-trained model that can learn comprehensive molecular representations and excel across a wide range of drug discovery tasks.

Methods

The core of the PanGu Drug Model is a novel graph-to-sequence asymmetric conditional variational autoencoder (cVAE) architecture. This approach allows the model to learn from two different molecular representations:

  • Input: 2D undirected cyclic graphs of molecules.
  • Output: Corresponding chemical formula strings.

This "Molecular Structure Formulating Transformation" (MSFT) mimics how humans learn to translate between visual structures and textual descriptions. Key methodological innovations include:

  • Layered Latent Space: Inspired by ECFP fingerprints, PanGu employs a layered latent space that captures molecular information at different "radius scales," providing a more expressive and interpretable representation.
  • Asymmetric Architecture: Unlike traditional autoencoders, PanGu uses an asymmetric graph-to-sequence architecture, avoiding the computational complexity of graph isomorphism matching for reconstruction.
  • Conditional Variational Autoencoder (cVAE): This allows for guided generation and optimization by incorporating desired molecular properties into the decoding process.

PanGu is pre-trained on a massive dataset of 1.7 billion molecules using self-supervised learning. For downstream tasks, it can be used in two ways:

  1. Fine-tuning: The encoder is integrated with a task-specific predictor, and the entire model is fine-tuned.
  2. PanGu Fingerprint: The latent space representation (including the condition vector) is extracted and used as a fixed-size molecular descriptor, with a simple MLP predictor.

Datasets

PanGu was pre-trained on a massive dataset comprising approximately 1.7 billion SMILES strings from:

  • ZINC20
  • DrugSpaceX
  • UniChem

For evaluation, the model was tested on various benchmark datasets for different drug discovery tasks, including:

  • Molecule Property Prediction: Datasets from MoleculeNet (e.g., BBBP, SIDER, ClinTox, BACE, Tox21, ToxCast, FreeSolv, ESOL, Lipophilicity, QM7, QM8).
  • Compound-Protein Interaction (CPI) Prediction: Human, C. elegans, and BindingDB datasets.
  • Drug-Drug Interaction (DDI) Prediction: A dataset containing 192,284 DDIs.
  • Chemical Reaction Productivity Prediction: A dataset with 3,956 chemical reactions.

The paper also generated a novel dataset of 100 million drug-like small molecules using the PanGu molecule generator.

Key Insights and Ideas

  • Human-like Learning: The MSFT approach, translating molecular graphs to sequences, effectively mimics human learning and provides a powerful way to learn molecular representations.
  • Layered Latent Space is Key: The layered latent space design significantly enhances the model's expressive power, outperforming models without this feature in molecule property prediction tasks. This suggests that capturing multi-scale structural information is crucial for robust molecular representation.
  • Unified Model Power: PanGu demonstrates that a single pre-trained model can achieve state-of-the-art performance across a broad spectrum of drug discovery tasks, from prediction to generation and optimization, highlighting the potential for a unified AI platform in drug R&D.
  • PanGu Fingerprint as an Efficient Alternative: The PanGu Fingerprint offers a computationally efficient alternative to fine-tuning, providing competitive results while significantly reducing overhead, making it a practical choice for many prediction tasks.
  • Generative Powerhouse: The ability to generate 100 million novel molecules with high novelty (99.68%) opens up new avenues for exploring uncharted chemical space and supplementing existing drug databases.

Key Results

  • State-of-the-Art Performance: PanGu achieved state-of-the-art results on 20 drug discovery tasks, including molecule property prediction, molecule generation, and molecule optimization.
  • Molecule Property Prediction:
    • Fine-tuned PanGu outperformed existing deep learning models in all 11 property prediction tasks, showing an average improvement of 0.8% in classification accuracies and a 4.9% decrease in regression errors compared to GROVER.
  • PanGu Fingerprint outperformed ECFP on all tasks, with an average improvement of 8.0% in classification accuracy and a 37.2% decrease in regression error.
  • PanGu Fingerprint ranked first in ROC-AUC, precision, and recall on human and C. elegans CPI prediction datasets.
  • Molecule Generation: PanGu generated a library of 100 million drug-like small molecules with 99.68% novelty, demonstrating high validity (nearly 100%) and uniqueness (99.95%).
  • Molecule Optimization: PanGu significantly improved target properties (e.g., penalized logP) by up to 4.8 times that of JT-VAE, even under strict similarity constraints. For example, with a similarity constraint of $\delta=0.0$, PanGu improved the property by 9.13 on average, compared to JT-VAE's 1.91.
  • Efficiency: PanGu was pre-trained on a cluster of 256 Ascend 910 AI processors for 10 epochs within 15 days, showcasing efficient training on a large scale.

Limitations and Future Work

  • Data Availability: While PanGu was trained on a massive dataset, the paper notes that some property records only cover a few chemicals, leading to limited labeled data for certain tasks.
  • Computational Overhead: While PanGu Fingerprint offers efficiency, fine-tuning PanGu still requires significant computational resources.
  • Similarity vs. Property Improvement: The paper acknowledges that there's an inherent trade-off between achieving higher property improvement and maintaining high molecular similarity during optimization.
  • Future Work: The authors suggest that the PanGu model can be further integrated with other properties for fine-tuning and that the web application for molecule optimization can be expanded. They also believe PanGu can serve as a foundation for future pre-training models in AI drug discovery.

Summary updated

Abstract

Abstract Recent achievements in large-scale pre-trained models like GPT-3 and PanGu-α showed amazing performances in many downstream tasks, which makes AI friendlier toward industrial users. Deep learning has been recognized as the most promising technology for pharmaceuticals, a powerful molecule pre-trained model could save researchers tons of time. In chemistry classes, the students learn two molecule representations, the molecular formula and the structure formula, and learn to translate them from one way to the other. Inspired by this, we developed a novel deep learning architecture using a graph-to-sequence asymmetric conditional variational autoencoders, called PanGu Drug Model, which can appropriately characterize the molecule from both representations and improve the performance of downstream drug discovery tasks. After pretrained with 1.7 billion small molecules, our proposed model achieved the state-of-the-art results in 20 drug discovery tasks, such as molecule property prediction (predict ADMET properties, compound-target interactions, drug-drug interactions and chemical reaction productivity), molecule generation and molecule optimization. A new drug screening library of 100 million drug-like small molecules with 99.68% novelty was generated by PanGu molecule generator, which could efficiently produce novel compounds with similar physiochemical properties to given distribution, this library could be used to supplement existing compound databases. In addition, PanGu molecule optimizer could optimize the chemical structures of starting molecule with improved molecular property of interest. An automatic multi-objective optimization web application implemented by PanGu Drug Model is available at http://www.pangu-drug.com/ .
BibTeX
@misc{lin2022pangu,
  author = {Xinyuan Lin and Chi Xu and Zhaoping Xiong and Xinfeng Zhang and Ningxi Ni and Bolin Ni and Jianlong Chang and Ruiqing Pan and Zidong Wang and Fan Yu and Qi Tian and Hualiang Jiang and Mingyue Zheng and Nan Qiao},
  title = {{PanGu Drug Model: Learn a Molecule Like a Human}},
  year = {2022},
  doi = {10.1101/2022.03.31.485886},
  howpublished = {bioRxiv},
}
Figure