Abstract
Molecular language modeling has traditionally relied on generic Masked Language Modelling (MLM) tasks, which often fail to capture the underlying chemical logic of SMILES strings. We present SmilesT5, a framework that treats molecular property prediction as a text-to-text problem using an encoder-decoder architecture. By introducing domain-specific pretraining tasks—Scaffold Reconstruction and Fragment Identification—SmilesT5 achieves state-of-the-art performance on MoleculeNet benchmarks. Notably, these chemistry-aware tasks significantly improve data efficiency, allowing models trained on 1 million molecules to rival those trained on 100 million.
Introduction
In drug discovery, the scarcity of labeled data for molecular properties like toxicity or permeability necessitates effective self-supervised pretraining. While Masked Language Modelling (MLM) has been the standard for pretraining molecular transformers, it is often "chemistry-blind," focusing on character-level syntax rather than structural relationships. The authors argue that predicting a missing character in a SMILES string does not guarantee an understanding of 3D structure or chemical behavior. SmilesT5 addresses this by forcing the model to learn the "grammar" of chemistry through structural reconstruction and functional group identification.
Methods
The SmilesT5 architecture utilizes the Text-to-Text Transfer Transformer (T5) framework. Unlike encoder-only models, the encoder-decoder structure allows for flexible input-output mappings. The authors introduced two primary domain-specific pretraining objectives:
- Scaffold Reconstruction: The model identifies and reconstructs the Murcko scaffold (the core ring system) from a full SMILES string.
- Fragment Identification: The model identifies 86 specific functional groups (e.g., benzene, carboxylic acid) within the molecule.
These tasks are implemented using T5's prefix system, enabling multi-task learning by prepending specific prompts (e.g., scaffold: or fragments:) to the input SMILES string. The models were pretrained on subsets of the ZINC15 dataset (1M and 100M molecules).
Results
Evaluation across six MoleculeNet classification benchmarks (BBBP, BACE, HIV, Clintox, SIDER, and Tox21) demonstrates that domain-specific pretraining consistently outperforms standard MLM.
Key findings include:
- Superiority of Chemistry-Aware Tasks: The combination of Scaffold and Fragment tasks statistically outperformed MLM across all benchmarks, as confirmed by Mann-Whitney U tests.
- Data Efficiency: Models pretrained on 1 million molecules using these tasks often matched or exceeded the performance of 100-million-molecule MLM models, suggesting that task quality can compensate for data quantity.
- State-of-the-Art Performance: The SmilesT5 XL variant (2.2B parameters) surpassed existing models such as MolFormer and ChemBERTa-2 in nearly every benchmark.
- Rich Embeddings: Frozen representations from the SmilesT5 encoder are sufficiently informative to allow simple classifiers like Random Forest to achieve competitive results, reducing the need for massive computational resources during fine-tuning.
Discussion and Conclusion
SmilesT5 demonstrates that teaching a model what a molecule is made of is more effective than simple character-level prediction. While the benefits of domain-specific pretraining are most pronounced in low-data regimes, the model's ability to reconstruct unseen scaffolds with 98.7% accuracy indicates a deep understanding of molecular structure. Future work will explore expanding the task library to include 3D geometry and quantum properties, further bridging the gap between language modeling and physical chemistry.
Summary updated