Retrosynthesis with Attention-Based NMT Model and Chemical Analysis of the "Wrong" Predictions

Hongliang Duan, Ling Wang, Chengyun Zhang, Jianjun Li

Abstract

We cast retrosynthesis as a machine translation problem by introducing a special Tensor2Tensor (T2T) model, an entirely attention-based and fully data-driven architecture. Given a dataset comprising approximately 50,000 diverse reactions extracted from USPTO patents, the model significantly outperforms the baseline sequence-to-sequence (seq2seq) model (37.4%) by achieving a top-1 accuracy of 54.1%. To optimize performance, parameters such as batch size and training time were thoroughly investigated. Additionally, we offer a novel insight into the causes of grammatically invalid SMILES and conduct a test in which experienced chemists analyze "wrong" predictions. We find that many are chemically plausible despite differing from the ground truth, suggesting the model's effectiveness is underestimated and its "true" top-1 accuracy can reach 64.6%.

Introduction

Retrosynthesis—the process of identifying starting materials for a target molecule—is a cornerstone of organic chemistry. Traditionally, this relied on rule-based systems like LHASA, which require manual encoding of thousands of chemical rules by experts. These systems are expensive to maintain and often fail to generalize to novel chemistry.

Recent approaches have treated retrosynthesis as a Neural Machine Translation (NMT) task using SMILES (Simplified Molecular Input Line Entry System) notation. However, earlier Recurrent Neural Network (RNN) based seq2seq models faced two major hurdles: a high rate of grammatically invalid SMILES strings and a rigid evaluation metric that penalizes valid alternative synthetic pathways not present in the training database.

Methodology

This study implements the Tensor2Tensor (T2T) model, a Transformer architecture that replaces recurrence with self-attention mechanisms. This allows the model to weigh the importance of different atoms in a SMILES string simultaneously, capturing long-range structural dependencies more effectively than LSTMs.

Model Configuration

  • Encoder-Decoder Stack: Utilizes multi-head attention to process molecular structures.
  • Positional Encoding: Employs trigonometric functions to maintain spatial/order information of atoms.
  • Optimization: Performance was maximized by using large batch sizes (up to 8192 tokens) and checkpoint averaging of the final 20 saved models.

Human-in-the-Loop Evaluation

To address the limitations of standard accuracy metrics, 10 experienced chemists reviewed predictions labeled as "wrong" by the automated system. They evaluated these predictions for chemical plausibility, checking if the suggested reactants could realistically form the target product via known mechanisms.

Key Results

The T2T model demonstrated a substantial performance leap over traditional RNN baselines. As shown in the accuracy comparison, the Transformer architecture consistently outperformed the baseline across all Top-k metrics.

Comparison of Top-k accuracy between the baseline seq2seq model and the Transformer-based T2T model.
  • Top-1 Accuracy: 54.1% (vs. 37.4% for baseline).
  • Top-10 Accuracy: 70.1%.
  • Invalidity Rate: Reduced to 3.4% (from >10% in previous models).

Chemical Analysis of "Wrong" Predictions

A core finding of this research is the "Ground Truth Fallacy." Standard evaluation metrics assume there is only one correct way to synthesize a molecule. However, chemical space allows for multiple valid routes.

Illustration of the T2T model workflow and a comparison between a "wrong" prediction (chemically plausible) and the database ground truth.

By analyzing 5,004 test samples, the authors found that 10.5% of the model's "errors" were actually chemically valid alternative pathways (e.g., using different coupling partners or oxidation states). When these plausible reactions are included, the "true" top-1 accuracy rises to 64.6%.

Grammatically invalid SMILES were found to be primarily caused by structural complexity, such as polycyclic or bridged ring systems, where the model lacks sufficient training examples to master the specific string syntax required for closure.

Conclusion

The application of the Transformer-based T2T model to retrosynthesis represents a significant advancement in data-driven chemistry. By moving beyond simple string matching and incorporating chemical expertise into the evaluation, this work demonstrates that NMT models are far more capable of "understanding" chemical logic than previously thought. Future work will focus on expanding the dataset for complex cyclic structures and extending the model to multi-step synthetic planning.

Summary updated

Abstract

We cast retrosynthesis as a machine translation problem by introducing a special Tensor2Tensor, an entire attention-based and fully data-driven model. Given a data set comprising about 50,000 diverse reactions extracted from USPTO patents, the model significantly outperforms seq2seq model (34.7%) on a top-1 accuracy by achieving 54.1%. For yielding better results, parameters such as batch size and training time are thoroughly investigated to train the model. Additionally, we offer a novel insight into the causes of grammatically invalid SMILES, and conduct a test in which experienced chemists pick out and analyze the "wrong" predictions that may be chemically plausible but differ from the ground truth. Actually, the effectiveness of our model is un-derestimated and the "true" top-1 accuracy can reach to 64.6%.
BibTeX
@misc{duan2019retrosynthesis,
  author = {Hongliang Duan and Ling Wang and Chengyun Zhang and Jianjun Li},
  title = {{Retrosynthesis with Attention-Based NMT Model and Chemical Analysis of the "Wrong" Predictions}},
  year = {2019},
  eprint = {1908.00727},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.00727},
}
Figure