Abstract
Deep reinforcement learning methods have been shown to be potentially powerful tools for de novo design. Recurrent neural network (RNN)-based techniques are the most widely used methods in this space. In this work, we examine the behaviour of RNN-based methods when there are few (or no) examples of molecules with the desired properties in the training data. We find that targeted molecular generation is often possible, but the diversity of generated molecules is often reduced, and it is not possible to control the composition of generated molecular sets. To help overcome these issues, we propose a new curriculum learning-inspired, recurrent Iterative Optimisation Procedure (rIOP) that enables the optimisation of generated molecules for seen and unseen molecular profiles and allows the user to control whether a molecular profile is explored or exploited. Using our method, we generate specific and diverse sets of molecules with up to 18 times more scaffolds than standard methods for the same sample size. However, our results also point to significant limitations of one-dimensional molecular representations as used in this space. We find that the success or failure of a given molecular optimisation problem depends on the choice of SMILES.
Introduction: The Challenge of Molecular Extrapolation
In drug discovery, the objective is frequently to identify molecules with specific, often extreme, property profiles—the "needle in a haystack." Generative AI models, particularly Recurrent Neural Networks (RNNs) utilizing SMILES (Simplified Molecular Input Line Entry System) strings, have become standard tools for this task. However, these models face two significant challenges: extrapolation and mode collapse. Models often struggle to generate molecules with properties entirely absent from their training data, and when pushed to optimize via Reinforcement Learning (RL), they tend to produce a narrow set of nearly identical structures that "cheat" the reward function to maximize scores without providing chemical variety.
Methods: The recurrent Iterative Optimisation Procedure (rIOP)
To address these limitations, the researchers developed the recurrent Iterative Optimisation Procedure (rIOP). This method is built upon the REINVENT framework, which employs an RNN "Prior" (trained on general chemical space) and an "Agent" that is fine-tuned using RL to maximize a specific property reward.
The core innovation of rIOP is the use of curriculum learning. Instead of attempting to reach an extreme property target in a single step, rIOP breaks the task into a sequence of increasingly difficult intermediate goals. By using step-wise optimization, the model first learns to achieve a moderate property value. Once successful, that model becomes the starting point for the next, more challenging step. This prevents the model from becoming "lazy" and sticking to the first few successful structures it finds.
Key Results: Diversity and Control
The curriculum approach significantly improves the model's ability to explore chemical space. For example, when optimizing for high Topological Polar Surface Area (TPSA), the rIOP method demonstrated a clear, step-wise shift in the generated distribution across successive training iterations.
Major findings from the study include:
- Massive Diversity Gains: In TPSA optimization tasks, rIOP produced 18 times more unique scaffolds (55 vs. 3) than standard REINVENT for the same sample size.
- Extrapolation Success: The study demonstrated that RL models can successfully find molecules with properties that had 0% representation in the initial training set, provided the underlying chemical logic remains consistent with the training data.
- The SMILES Bottleneck: A critical finding was the dependency on SMILES representation. Because a single molecule can have multiple valid SMILES strings, the model's success often depended on the specific "spelling" used, highlighting a fundamental weakness in 1D representations.
Conclusion
While rIOP offers a powerful way to generate diverse and specific molecular libraries, the research underscores the limitations of 1D SMILES strings. Future work may need to transition toward more robust representations, such as SELFIES or graph-based models, to fully realize the potential of de novo molecular design and overcome the inconsistencies inherent in text-based chemical strings.
Summary updated