Optimization of binding affinities in chemical space with generative pre-trained transformer and deep reinforcement learning

Xiaopeng Xu, Juexiao Zhou, Chen Zhu, Qing Zhan, Zhongxiao Li, Ruochi Zhang, Yu Wang, Xingyu Liao, Xin GaoF1000Research 2024 · DOI

TL;DR

This paper introduces SGPT-RL, a novel method that combines a Generative Pre-trained Transformer (GPT) with deep reinforcement learning (RL) to optimize molecular binding affinities. SGPT-RL demonstrates superior performance in generating molecules with high binding affinity, particularly for targets with limited known active compounds, showcasing its potential for accelerating drug discovery.

What Problem Does This Paper Address?

Discovering new drug candidates with desirable properties, especially high binding affinity to a specific biological target, is a major challenge in the pharmaceutical industry. The sheer vastness of chemical space (estimated at 10^23 to 10^60 molecules) makes exhaustive screening impossible. Traditional methods like high-throughput screening and in vitro assays are time-consuming and expensive. While computational methods like molecular docking and Quantitative Structure-Activity Relationship (QSAR) models help, they are often limited to searching within existing compound libraries. This paper addresses the need for de novo molecular design methods that can efficiently explore this vast chemical space and generate novel molecules with optimized binding affinities, going beyond pre-existing libraries.

Methods

The core innovation of this work is the SGPT-RL framework, which integrates a Generative Pre-trained Transformer (GPT) as the policy network within a reinforcement learning (RL) agent.

  • Generative Pre-trained Transformer (GPT) as Prior Model: A simplified GPT model (around 6M parameters) is first trained on a large dataset of lead-like molecules (from the Moses benchmark) to learn the general chemical space and molecular properties. This "prior model" captures the distribution of valid and chemically sensible molecules.
  • Reinforcement Learning (RL) Agent: The trained GPT model is then used to initialize an RL agent. This agent learns to generate molecules by iteratively predicting the next token (part of a SMILES string).
  • Optimization Objective: The RL agent is trained to optimize specific properties, such as binding affinity to a target. This is achieved by:
  • Generating a batch of molecules.
  • Scoring these molecules using predefined functions (e.g., QSAR models for DRD2 activity, molecular docking for ACE2 affinity).
  • Combining the target score with the prior likelihood of generating that molecule (from the GPT model) to create an "augmented likelihood."
    • Using the difference between the augmented likelihood and the agent's own likelihood as the loss function to update the agent's policy. This process biases the agent towards generating molecules with higher target scores while maintaining chemical validity.
  • Baseline Comparison: The popular Reinvent method, which uses a Gated Recurrent Unit (GRU) as its policy network, is used as a baseline for comparison.

Datasets

  • Moses Benchmark: Used for training the GPT prior model. This dataset contains approximately 1.9 million lead-like molecules from the Zinc database.
  • ExCAPE-DB: Used to obtain known active molecules for the target proteins. For DRD2, 8,036 active molecules were retrieved, and for ACE2, 56 active molecules were retrieved. These were not found in the Moses training dataset.

Key Insights and Ideas

  • GPT as a Powerful Policy Network: The study demonstrates that a GPT, known for its success in natural language processing, can be effectively adapted as a policy network for molecular generation, outperforming traditional RNN-based approaches like GRU in certain aspects.
  • Bridging Chemical Space and Binding Affinity: SGPT-RL successfully bridges the gap between learning general chemical space distributions and optimizing for specific binding affinities, a crucial step in drug discovery.
  • Scaffold Exploration and Pattern Learning: SGPT-RL exhibits a more systematic exploration of chemical scaffolds compared to Reinvent, gradually increasing molecular complexity (e.g., number of rings) and identifying conserved patterns, which is valuable for chemists.
  • Applicability to Low-Data Scenarios: The method shows promise for targets with few known active molecules (like ACE2), where traditional QSAR models are less effective, relying instead on molecular docking.

Key Results

  • Moses Benchmark: The SGPT-RL prior model achieved good validity (0.936), uniqueness (0.997), and novelty (0.946).
  • DRD2 Task (QSAR-based): Both SGPT-RL and Reinvent generated molecules with good DRD2 activities. SGPT-RL explored more scaffolds and showed continuous improvement over more steps than Reinvent.
  • ACE2 Task (Docking-based): SGPT-RL significantly outperformed Reinvent, achieving superior docking scores. This is particularly noteworthy as ACE2 is a target with limited known active compounds. SGPT-RL also identified specific patterns, such as double-ring structures, in the generated molecules.
  • Scaffold Analysis: SGPT-RL showed a tendency to generate molecules with more rings and conserved scaffold patterns, especially in the ACE2 task, suggesting a more directed exploration strategy.

Limitations and Future Work

  • Dataset Bias: The prior models were trained on the Moses dataset, which primarily consists of lead-like molecules. This bias might limit the exploration of molecules outside this space, although it could be beneficial for generating molecules with desirable drug-like properties. The authors suggest using other training data for prior models if exploring beyond lead-like molecules is desired.
  • Docking Experiment Scope: The study focused on ACE2 docking. The authors suggest that extending docking experiments to additional targets would further validate their findings.
  • Computational Resources: The paper mentions the need for high-end GPUs with large memory due to the memory-intensive nature of transformer decoders generating batches of molecules during RL.
  • Further Improvements: The authors suggest that the "repetitive explanations of abbreviations" and "fluency of the text" could be improved. They also suggest incorporating figures demonstrating property distribution evolution into the main content for better clarity.

Summary updated

Abstract

The superior performance of SGPT-RL in the ACE2 task indicates that it can be applied to the virtual screening process where molecular docking is widely used as the criteria. Besides, the scaffold patterns learned by SGPT-RL during the exploration process can assist chemists to better design and discover novel lead candidates.
BibTeX
@inproceedings{xu2024optimization,
  author = {Xiaopeng Xu and Juexiao Zhou and Chen Zhu and Qing Zhan and Zhongxiao Li and Ruochi Zhang and Yu Wang and Xingyu Liao and Xin Gao},
  title = {{Optimization of binding affinities in chemical space with generative pre-trained transformer and deep reinforcement learning}},
  booktitle = {F1000Research},
  year = {2024},
  doi = {10.12688/f1000research.130936.2},
}
Figure