Imagine evaluating a chess-playing AI by checking whether its moves exactly match a single historical grandmaster game. If the AI finds a brilliant, alternative checkmate sequence that wasn't played in that specific match, the evaluation framework scores it as a failure.
For years, computer-assisted synthesis planning (CASP) and retrosynthesis models have suffered from a similar constraint known as the "ground-truth dogma." Models are typically evaluated using exact-match metrics (such as Top-1 or Top-$k$ accuracy) against a single reference reaction extracted from historical patent databases. However, chemistry is fundamentally open-ended. This rigid evaluation paradigm creates a severe disconnect between high benchmark scores and actual laboratory utility.
The Failure Modes of Exact Match
The reliance on exact string or graph matching introduces several systemic biases that hinder the development of practical retrosynthesis tools.
First, exact-match metrics penalize valid chemical creativity. As Chaochao Yan et al. observe in RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist, a single target molecule can often be reached through multiple valid chemical pathways. When an evaluation framework demands an exact match with a single ground-truth reference, it penalizes models for proposing chemically sound alternative solutions that happen to differ from the historical record. This limitation is further echoed by Songtao Liu et al. in FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning, who argue that set-wise exact matches are too restrictive to recognize legitimate synthesis paths missing from reference datasets.
Second, exact matching rewards memorization over chemical intuition. Traditional top-$N$ metrics incentivize models to replicate specific documented patent routes rather than learning the underlying principles of chemical viability (Philippe Schwaller et al., Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy). Consequently, a model might achieve high accuracy on individual reaction templates but fail completely when trying to chain them into a viable multi-step synthetic route (Amol Thakkar et al., Datasets and their influence on the development of computer assisted synthesis planning tools in the pharmaceutical domain; Hongyu Tu et al., Retrosynthesis Prediction Revisited).
Third, these metrics ignore critical chemical nuances. Traditional accuracy metrics treat all prediction errors as equal, failing to capture whether a mistaken prediction is a minor stereochemical issue or a completely impossible transformation (Victor Sabanza Gil et al., Holistic chemical evaluation reveals pitfalls in reaction prediction models). Furthermore, because historical databases are inherently noisy and incomplete, a model might propose a highly creative, valid, or even superior alternative protocol that is marked as a failure simply because it is unrecorded (Ziqi Chen et al., RLSynC: Offline-Online Reinforcement Learning for Synthon Completion). As summarized by Peilin Zhao et al. in A Unified View of Deep Learning for Reaction and Retrosynthesis Prediction: Current Status and Future Challenges, this rigid approach actively discourages model diversity and limits real-world utility.
Shifting to Plausibility and Diversity Metrics
To bridge the gap between benchmark performance and laboratory reality, researchers are shifting toward evaluation frameworks that prioritize chemical plausibility and pathway diversity over literal string matching.
Instead of treating the "published" reaction as the absolute truth (Min Htoo Lin et al., Improving the performance of models for one-step retrosynthesis through re-ranking), newer frameworks evaluate whether a predicted reaction is thermodynamically and kinetically feasible. For example, the ChemCensor framework, introduced by Bogdan Zagribelnyy et al. in When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs, explicitly replaces exact-match metrics with plausibility-based scoring to accommodate the open-ended nature of chemical discovery.
This shift allows models to be fairly evaluated on complex aspects of synthesis that exact matching completely misses:
- Reaction Conditions and Byproducts: Models must be assessed on their understanding of the entire reaction environment, including solvents, catalysts, and plausible side products (Sean Current et al., DiffER: Categorical Diffusion Models for Chemical Retrosynthesis; Qingyu Wang et al., Towards Knowledge‑and‑Data‑Driven Organic Reaction Prediction: RAG‑Enhanced and Reasoning‑Powered Hybrid System with LLMs). For instance, the UniCon framework developed by Hyukjun Lim et al. in Learning Reaction-Condition Plausibility for Evaluation and Self-Curation Under Noisy and Non-Unique Supervision learns to score the viability of reaction-condition pairs directly, enabling the identification of superior alternative protocols that deviate from historical records.
- Exploration and Round-Trip Verification: Rather than restricting models to limited training datasets, the authors of newer frameworks leverage generative mechanisms to explore a wider space of feasible reactions, utilizing metrics like "round-trip accuracy" (verifying a retrosynthetic step by running it through a forward reaction prediction model) to better reflect real-world synthesis success (Piotr Gaiński et al., Diverse and feasible retrosynthesis using GFlowNets).
By evaluating models based on whether a reaction can work rather than whether it has been written down exactly this way before, plausibility-based evaluation transforms retrosynthesis from a rigid text-memorization task into a true tool for scientific discovery.
Key Foundational Work
RetroXpert: Decompose Retrosynthesis Prediction Like A Chemist (2020) — contributed 3 concept elements to this direction.
Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy (2020) — contributed 3 concept elements to this direction.
Datasets and their influence on the development of computer assisted synthesis planning tools in the pharmaceutical domain (2019) — contributed 2 concept elements to this direction.