Abstract
Computer-Assisted Synthesis Planning (CASP) tools can generate vast numbers of retrosynthesis routes, yet many are chemically implausible or redundant. This paper introduces Retro-BLEU, a novel metric adapted from natural language processing (NLP) that evaluates the quality of synthetic routes by analyzing sequences of reaction templates. By quantifying the overlap of template bigrams between proposed routes and historical patent data, Retro-BLEU provides a statistically grounded measure of chemical plausibility, effectively distinguishing realistic synthesis paths from AI-generated "hallucinations."
Introduction: The Plausibility Gap in CASP
Computer-Assisted Synthesis Planning (CASP) has revolutionized drug discovery by enabling the rapid generation of step-by-step instructions to build complex molecules. However, a significant gap remains between algorithmic output and laboratory feasibility. Many AI-generated routes include steps that fail due to selectivity issues or contain "redundant" transformations—such as unnecessary protection-deprotection cycles—that add length without synthetic value.
Existing metrics, such as route length or internal model confidence, often fail to capture these nuances. Shorter routes are not inherently better if they are chemically impossible, and model confidence frequently reflects training biases rather than real-world success.
Methodology: From Sentences to Synthesis
The core innovation of Retro-BLEU is the analogy between a chemical synthesis route and a sentence in human language. Just as certain words follow others to create coherent meaning, specific chemical reactions follow others to form a logical synthetic strategy.
Reaction Templates and n-gram Analysis
Instead of analyzing specific molecules, the authors utilize reaction templates—generalized rules describing core chemical transformations. Routes are decomposed into sequences of $n$ consecutive templates. The researchers found that bigrams ($n=2$) offer the optimal balance between capturing chemical context and maintaining statistical significance.
The Retro-BLEU Score
The metric is calculated based on two primary factors:
- Bigram Overlap: The fraction of template pairs in a generated route that appear in a reference database of "known-good" routes (e.g., USPTO patent data).
- Length Penalty: A factor that penalizes routes significantly longer than typical successful patent routes (with a target length $L=3$).
If a proposed route utilizes sequences of reactions that have been historically successful, it is statistically more likely to be chemically plausible.
Validation and Datasets
Retro-BLEU was validated using high-quality benchmarks, including PaRoutes (approx. 450,000 experimentally validated routes from US patents) and Retro*-190. The researchers generated millions of routes using state-of-the-art search algorithms like MCTS and Retro* to test the metric's ability to distinguish "real" patent routes from "fake" AI-generated ones.
Key Insights
- Chemical Coherence: Nearly 50% of reaction bigrams in new patent routes already exist in historical databases, suggesting that successful synthetic strategies follow predictable patterns.
- Identifying Hallucinations: Retro-BLEU successfully identifies "negative bigrams"—sequences common in AI routes but absent in patents, often representing redundant or circular transformations.
- Quality Over Brevity: The metric correctly ranks longer, logically sound patent routes higher than shorter, chemically impossible AI suggestions.
Results and Performance
Retro-BLEU consistently outperformed traditional baselines, such as reaction cost and cumulative probability, in identifying "true" patent routes among hundreds of alternatives. It demonstrated superior robustness and differentiating power, particularly in identifying "searchable routes" where individual steps are known but the overall sequence is flawed.
Limitations and Future Work
While powerful, Retro-BLEU has limitations:
- Innovation Penalty: Relying on historical data may penalize truly novel synthetic strategies.
- Database Dependency: The metric's accuracy depends on the quality and currency of the reference route database.
- Statistical vs. Physical: Retro-BLEU is a measure of statistical plausibility, not a definitive proof of chemical feasibility based on first principles.
Future research aims to integrate Retro-BLEU directly into CASP search processes to prune illogical synthetic branches in real-time.
Summary updated