Abstract
Transformer-based models trained on large and general purpose datasets consisting of molecular strings have recently emerged as a powerful tool for successfully modeling various structure-property relations. Inspired by this success, we extend the paradigm of training chemical language transformers on large-scale chemical datasets to generative tasks in this work. Specifically, we propose GP-MoLFormer, an autoregressive molecular string generator that is trained on more than 1.1B (billion) chemical SMILES. GP-MoLFormer uses a 46.8M parameter transformer decoder model with linear attention and rotary positional encodings as the base architecture. GP-MoLFormer's utility is evaluated and compared with that of existing baselines on three different tasks: de novo generation, scaffold-constrained molecular decoration, and unconstrained property-guided optimization. While the first two are handled with no additional training, we propose a parameter-efficient fine-tuning method for the last task, which uses property-ordered molecular pairs as input. We call this new approach pair-tuning. Our results show GP-MoLFormer performs better or comparable with baselines across all three tasks, demonstrating its general utility for a variety of molecular generation tasks. We further report strong memorization of training data in GP-MoLFormer generations, which has so far remained unexplored for chemical language models. Our analyses reveal that training data memorization and novelty in generations are impacted by the quality and scale of the training data; duplication bias in training data can enhance memorization at the cost of lowering novelty. We further establish a scaling law relating inference compute and novelty in generations.
Introduction
The "chemical space"—the set of all possible small molecules—is estimated to contain $10^{60}$ candidates. Navigating this vast landscape to find drugs with specific properties is a significant challenge. While Large Language Models (LLMs) have revolutionized natural language processing, their application to chemistry has often been limited by smaller datasets (usually 1–100 million molecules) and a lack of understanding regarding model memorization. If a model simply reproduces molecules from its training set, it fails to discover truly novel chemical entities. GP-MoLFormer addresses these challenges by building a generative foundation model capable of diverse tasks—from de novo design to scaffold decoration—while providing a rigorous analysis of how data scale and duplication affect the novelty of generated molecules.
Methodology
GP-MoLFormer utilizes a decoder-only Transformer architecture with 46.8 million parameters. To manage the computational complexity of long molecular strings, the model incorporates several key architectural features:
- Linear Attention: An efficient attention mechanism that scales linearly with sequence length, improving performance over standard quadratic attention.
- Rotary Positional Encodings (RoPE): These encodings better capture the positional dependencies of atoms within a SMILES string.
- Pair-Tuning: This novel parameter-efficient fine-tuning method enables property-guided optimization. Instead of training on absolute property values, the model learns from pairs of molecules where one has a superior property. It learns a "soft prompt" (prefix tokens) to steer generations toward optimized regions of the chemical manifold.
Data and Training
The model was pre-trained on a massive corpus of 1.1 billion SMILES strings sourced from the ZINC and PubChem databases. To study the effects of data redundancy, the authors also developed GP-MoLFormer-UNIQ, a variant trained on a de-duplicated subset of 650 million unique molecules. Evaluation was conducted using the MOSES benchmark for de novo generation and the DRD2 (Dopamine receptor) dataset for targeted activity optimization.
Results and Analysis
The Memorization-Novelty Trade-off
Unlike smaller models that often claim near 100% novelty, GP-MoLFormer exhibits significant memorization (~32% novelty at standard scales). This is interpreted as the model accurately learning the real-world distribution of known chemistry. Duplication in public databases was found to boost memorization at the expense of novelty; training on the de-duplicated "UNIQ" dataset increased novelty by 5–8%.
Scaling Laws and Property Optimization
The study established a mathematical scaling law ($y = ae^{-bx}$) relating the number of generated molecules to the novelty of the set, showing that as inference compute increases, the probability of finding new molecules decays exponentially. Despite this, GP-MoLFormer achieved state-of-the-art results in property optimization tasks, outperforming baselines like MARS and GraphDF. In scaffold decoration for the DRD2 task, the model achieved a 4.58% hit rate, surpassing specialized models.
Conclusion
GP-MoLFormer demonstrates the power of large-scale foundation models in chemistry. While SMILES strings can be syntactically brittle, the model's ability to steer generations via pair-tuning and its performance across diverse tasks mark a significant step forward. Future work will address anthropogenic biases in training data and explore alternative molecular representations to further enhance the exploration of chemical space.
Summary updated