Improving De Novo Molecular Design with Curriculum Learning

Jeff Guo, Vendy Fialková, Juan Diego Arango, Christian Margreitter, Jon Paul Janet, Kostas Papadopoulos, Ola Engkvist, Atanas Patronov

Abstract

Reinforcement learning (RL) is a powerful paradigm that has gained popularity across multiple domains. However, applying RL may come at a cost of multiple interactions between the agent and the environment. This cost can be especially pronounced when the single feedback from the environment is slow or computationally expensive, causing extensive periods of nonproductivity. Curriculum learning (CL) provides a suitable alternative by arranging a sequence of tasks of increasing complexity with the aim of reducing the overall cost of learning. Here, we demonstrate the application of CL for drug discovery. We implement CL in the de novo design platform, REINVENT, and apply it on illustrative de novo molecular design problems of different complexity. The results show both accelerated learning and a positive impact on the quality of the output when compared to standard policy based RL. To our knowledge, this is the first application of CL for the purposes of de novo molecular design. The code is freely available at https://github.com/MolecularAI/Reinvent.

Introduction

In de novo drug design, Reinforcement Learning (RL) agents are tasked with navigating an immense chemical search space—estimated between $10^{23}$ and $10^{60}$ molecules—to identify candidates that satisfy complex Multi-Parameter Optimization (MPO) criteria. A significant bottleneck in standard RL is the "nonproductive period," where an agent samples low-quality molecules that receive zero reward. Without feedback, the agent cannot compute useful gradients, leading to wasted computational resources, particularly when using expensive physics-based docking simulations.

Methodology: The Curriculum Learning Framework

To address these challenges, researchers at AstraZeneca implemented a Curriculum Learning (CL) strategy within the REINVENT platform. CL breaks down a complex final goal into a sequence of manageable "stepping stone" tasks.

The Curriculum Learning architecture consists of a Curriculum Phase, where the agent masters increasingly difficult objectives, followed by a Production Phase for the final target.

The process is divided into two primary stages:

  1. Curriculum Phase: The agent encounters a series of objectives ($O_1, O_2, \dots, O_n$) of increasing difficulty. Progression to the next task is governed by a "Progression Criterion," ensuring the agent has mastered the current level before moving on.
  2. Production Phase: After "graduating" from the curriculum, the agent focuses on the final Production Objective.

The agent utilizes a Recurrent Neural Network (RNN) with three Long Short-Term Memory (LSTM) layers to generate molecules in SMILES format. To maintain chemical diversity, a "Diversity Filter" (DF) is employed to penalize the redundant generation of identical chemical scaffolds.

Key Insights and Results

The efficacy of CL was demonstrated through several complex molecular design tasks, most notably in scaffold construction and protein-ligand docking for PDK1 inhibitors.

Scaffold Decomposition

One of the most striking results involved the construction of a complex dihydro-pyrazoloquinazoline scaffold. While a baseline RL agent failed to generate the target scaffold even once within 2,000 epochs, the CL agent successfully mastered the structure by learning its components piece-by-piece.

Learning progression for a target scaffold. The sawtooth pattern illustrates the agent mastering intermediate structural components before achieving the final complex scaffold.

Accelerated Productivity in Docking

In docking experiments, CL agents achieved immediate productivity. By using 2D or 3D similarity as a proxy curriculum before attempting full docking, the agent's policy was "pre-aligned" to favorable chemical spaces. This eliminated the ~200-epoch "blind search" period observed in baseline RL models. Furthermore, CL produced a significantly higher yield of unique, high-scoring scaffolds—generating over 9,000 high-quality compounds in some scenarios compared to fewer than 1,000 from the baseline.

Discussion and Future Directions

While CL significantly accelerates learning and improves output quality, it introduces a trade-off between optimization speed and molecular diversity. Highly focused curricula can narrow the agent's search space. Future research will focus on automating curriculum design—which currently requires expert manual setup—and developing more sophisticated diversity filters to maintain a broad exploration of chemical space.

Conclusion

This study represents the first application of Curriculum Learning to de novo molecular design. By structuring the learning process, CL enables RL agents to solve optimization problems that were previously intractable, offering a more efficient path for discovering novel therapeutic leads.

Code Availability: The implementation is available as part of the REINVENT platform on GitHub: https://github.com/MolecularAI/Reinvent

Summary updated

Abstract

<title>Abstract</title> Reinforcement learning (RL) is a powerful paradigm that has gained popularity across multiple domains. However, applying RL may come at a cost of multiple interactions between the agent and the environment. This cost can be especially pronounced when the single feedback from the environment is slow or computationally expensive, causing extensive periods of nonproductivity. Curriculum learning (CL) provides a suitable alternative by arranging a sequence of tasks of increasing complexity with the aim of reducing the overall cost of learning. Here, we demonstrate the application of CL for drug discovery. We implement CL in the de novo design platform, REINVENT, and apply it on illustrative de novo molecular design problems of different complexity. The results show both accelerated learning and a positive impact on the quality of the output when compared to standard policy based RL. To our knowledge, this is the first application of CL for the purposes of de novo molecular design. The code is freely available at https://github.com/MolecularAI/Reinvent.
BibTeX
@article{guo2022improving,
  author = {Jeff Guo and Vendy Fialková and Juan Diego Arango and Christian Margreitter and Jon Paul Janet and Kostas Papadopoulos and Ola Engkvist and Atanas Patronov},
  title = {{Improving De Novo Molecular Design with Curriculum Learning}},
  journal = {Nat. Mach. Intell},
  year = {2022},
  doi = {10.1038/s42256-022-00494-4},
}
Figure