Testing the limits of SMILES-based de novo molecular generation with curriculum and deep reinforcement learning

Maranga Mokaya, Fergus Imrie, Willem P. van Hoorn, Aleksandra Kalisz, Anthony R. Bradley, Charlotte M. DeaneNat. Mac. Intell 2023 · DOI

Abstract

Deep reinforcement learning methods have been shown to be potentially powerful tools for de novo design. Recurrent neural network (RNN)-based techniques are the most widely used methods in this space. In this work, we examine the behaviour of RNN-based methods when there are few (or no) examples of molecules with the desired properties in the training data. We find that targeted molecular generation is often possible, but the diversity of generated molecules is often reduced, and it is not possible to control the composition of generated molecular sets. To help overcome these issues, we propose a new curriculum learning-inspired, recurrent Iterative Optimisation Procedure (rIOP) that enables the optimisation of generated molecules for seen and unseen molecular profiles and allows the user to control whether a molecular profile is explored or exploited. Using our method, we generate specific and diverse sets of molecules with up to 18 times more scaffolds than standard methods for the same sample size. However, our results also point to significant limitations of one-dimensional molecular representations as used in this space. We find that the success or failure of a given molecular optimisation problem depends on the choice of SMILES.

Introduction: The Challenge of Molecular Extrapolation

In drug discovery, the objective is frequently to identify molecules with specific, often extreme, property profiles—the "needle in a haystack." Generative AI models, particularly Recurrent Neural Networks (RNNs) utilizing SMILES (Simplified Molecular Input Line Entry System) strings, have become standard tools for this task. However, these models face two significant challenges: extrapolation and mode collapse. Models often struggle to generate molecules with properties entirely absent from their training data, and when pushed to optimize via Reinforcement Learning (RL), they tend to produce a narrow set of nearly identical structures that "cheat" the reward function to maximize scores without providing chemical variety.

Methods: The recurrent Iterative Optimisation Procedure (rIOP)

To address these limitations, the researchers developed the recurrent Iterative Optimisation Procedure (rIOP). This method is built upon the REINVENT framework, which employs an RNN "Prior" (trained on general chemical space) and an "Agent" that is fine-tuned using RL to maximize a specific property reward.

The REINVENT framework: A Prior model provides chemical logic, while an Agent is fine-tuned via a Reward/Scoring loop to generate targeted SMILES.

The core innovation of rIOP is the use of curriculum learning. Instead of attempting to reach an extreme property target in a single step, rIOP breaks the task into a sequence of increasingly difficult intermediate goals. By using step-wise optimization, the model first learns to achieve a moderate property value. Once successful, that model becomes the starting point for the next, more challenging step. This prevents the model from becoming "lazy" and sticking to the first few successful structures it finds.

Key Results: Diversity and Control

The curriculum approach significantly improves the model's ability to explore chemical space. For example, when optimizing for high Topological Polar Surface Area (TPSA), the rIOP method demonstrated a clear, step-wise shift in the generated distribution across successive training iterations.

Density plots showing the step-wise optimization of TPSA using SrIOP, where each successive step (1-4) pushes the distribution toward higher values.

Major findings from the study include:

  • Massive Diversity Gains: In TPSA optimization tasks, rIOP produced 18 times more unique scaffolds (55 vs. 3) than standard REINVENT for the same sample size.
  • Extrapolation Success: The study demonstrated that RL models can successfully find molecules with properties that had 0% representation in the initial training set, provided the underlying chemical logic remains consistent with the training data.
  • The SMILES Bottleneck: A critical finding was the dependency on SMILES representation. Because a single molecule can have multiple valid SMILES strings, the model's success often depended on the specific "spelling" used, highlighting a fundamental weakness in 1D representations.

Conclusion

While rIOP offers a powerful way to generate diverse and specific molecular libraries, the research underscores the limitations of 1D SMILES strings. Future work may need to transition toward more robust representations, such as SELFIES or graph-based models, to fully realize the potential of de novo molecular design and overcome the inconsistencies inherent in text-based chemical strings.

Summary updated

BibTeX
@article{mokaya2023testing,
  author = {Maranga Mokaya and Fergus Imrie and Willem P. van Hoorn and Aleksandra Kalisz and Anthony R. Bradley and Charlotte M. Deane},
  title = {{Testing the limits of SMILES-based de novo molecular generation with curriculum and deep reinforcement learning}},
  journal = {Nat. Mac. Intell.},
  year = {2023},
  doi = {10.1038/s42256-023-00636-2},
}

Loading related content...

Figure