Retro-BLEU: Quantifying Chemical Plausibility of Retrosynthesis Routes through Reaction Template Sequence Analysis

Junren Li, Lei Fang, Jian-Guang LouarXiv 2023 · DOI · arXiv

Abstract

Computer-Assisted Synthesis Planning (CASP) tools can generate vast numbers of retrosynthesis routes, yet many are chemically implausible or redundant. This paper introduces Retro-BLEU, a novel metric adapted from natural language processing (NLP) that evaluates the quality of synthetic routes by analyzing sequences of reaction templates. By quantifying the overlap of template bigrams between proposed routes and historical patent data, Retro-BLEU provides a statistically grounded measure of chemical plausibility, effectively distinguishing realistic synthesis paths from AI-generated "hallucinations."

Introduction: The Plausibility Gap in CASP

Computer-Assisted Synthesis Planning (CASP) has revolutionized drug discovery by enabling the rapid generation of step-by-step instructions to build complex molecules. However, a significant gap remains between algorithmic output and laboratory feasibility. Many AI-generated routes include steps that fail due to selectivity issues or contain "redundant" transformations—such as unnecessary protection-deprotection cycles—that add length without synthetic value.

Existing metrics, such as route length or internal model confidence, often fail to capture these nuances. Shorter routes are not inherently better if they are chemically impossible, and model confidence frequently reflects training biases rather than real-world success.

Methodology: From Sentences to Synthesis

The core innovation of Retro-BLEU is the analogy between a chemical synthesis route and a sentence in human language. Just as certain words follow others to create coherent meaning, specific chemical reactions follow others to form a logical synthetic strategy.

Reaction Templates and n-gram Analysis

Instead of analyzing specific molecules, the authors utilize reaction templates—generalized rules describing core chemical transformations. Routes are decomposed into sequences of $n$ consecutive templates. The researchers found that bigrams ($n=2$) offer the optimal balance between capturing chemical context and maintaining statistical significance.

The analogy between Machine Translation BLEU and Retrosynthesis planning Retro-BLEU.

The Retro-BLEU Score

The metric is calculated based on two primary factors:

  1. Bigram Overlap: The fraction of template pairs in a generated route that appear in a reference database of "known-good" routes (e.g., USPTO patent data).
  2. Length Penalty: A factor that penalizes routes significantly longer than typical successful patent routes (with a target length $L=3$).

If a proposed route utilizes sequences of reactions that have been historically successful, it is statistically more likely to be chemically plausible.

Validation and Datasets

Retro-BLEU was validated using high-quality benchmarks, including PaRoutes (approx. 450,000 experimentally validated routes from US patents) and Retro*-190. The researchers generated millions of routes using state-of-the-art search algorithms like MCTS and Retro* to test the metric's ability to distinguish "real" patent routes from "fake" AI-generated ones.

Key Insights

  • Chemical Coherence: Nearly 50% of reaction bigrams in new patent routes already exist in historical databases, suggesting that successful synthetic strategies follow predictable patterns.
  • Identifying Hallucinations: Retro-BLEU successfully identifies "negative bigrams"—sequences common in AI routes but absent in patents, often representing redundant or circular transformations.
  • Quality Over Brevity: The metric correctly ranks longer, logically sound patent routes higher than shorter, chemically impossible AI suggestions.

Results and Performance

Retro-BLEU consistently outperformed traditional baselines, such as reaction cost and cumulative probability, in identifying "true" patent routes among hundreds of alternatives. It demonstrated superior robustness and differentiating power, particularly in identifying "searchable routes" where individual steps are known but the overall sequence is flawed.

Limitations and Future Work

While powerful, Retro-BLEU has limitations:

  • Innovation Penalty: Relying on historical data may penalize truly novel synthetic strategies.
  • Database Dependency: The metric's accuracy depends on the quality and currency of the reference route database.
  • Statistical vs. Physical: Retro-BLEU is a measure of statistical plausibility, not a definitive proof of chemical feasibility based on first principles.

Future research aims to integrate Retro-BLEU directly into CASP search processes to prune illogical synthetic branches in real-time.

Summary updated

Abstract

Computer-assisted methods have emerged as valuable tools for retrosynthesis analysis. However, quantifying the plausibility of generated retrosynthesis routes remains a challenging task. We introduce Retro-BLEU, a statistical metric adapted from the well-established BLEU score in machine translation, to evaluate the plausibility of retrosynthesis routes based on reaction template sequences analysis. We demonstrate the effectiveness of Retro-BLEU by applying it to a diverse set of retrosynthesis routes generated by state-of-the-art algorithms and compare the performance with other evaluation metrics. The results show that Retro-BLEU is capable of differentiating between plausible and implausible routes. Furthermore, we provide insights into the strengths and weaknesses of Retro-BLEU, paving the way for future developments and improvements in this field.
BibTeX
@misc{li2023retro,
  author = {Junren Li and Lei Fang and Jian-Guang Lou},
  title = {{Retro-BLEU: Quantifying Chemical Plausibility of Retrosynthesis Routes through Reaction Template Sequence Analysis}},
  year = {2023},
  eprint = {2311.06304},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.06304},
}
Figure