RetroOOD: Understanding Out-of-Distribution Generalization in Retrosynthesis Prediction

Yemin Yu, Luotian Yuan, Ying Wei, Hanyu Gao, Xinhai Ye, Zhihua Wang, Fei Wu

TL;DR

RetroOOD provides a rigorous framework for understanding why retrosynthesis models fail when they encounter new molecules or reaction types. By disentangling covariate shift (changes in molecular structure) from label shift (changes in reaction strategies) and introducing new OOD benchmarks, the authors propose model-agnostic techniques that improve generalization by an average of 4.6%.

What Problem Does This Paper Address?

Retrosynthesis—the process of identifying the simpler building blocks needed to synthesize a complex molecule—is a cornerstone of organic chemistry. While machine learning models have become quite good at this on standard benchmarks, they often "break" when deployed in the real world.

The core issue is Out-of-Distribution (OOD) generalization. Most models are trained and tested on data that look similar (i.e., they are independent and identically distributed, or i.i.d.). However, a model trained on simple molecules might fail on complex ones (covariate shift), or a model might struggle to predict a reaction type it hasn't seen frequently, even if the target molecule is familiar (label shift). Before this paper, there was no systematic way to measure or mitigate these specific types of shifts in retrosynthesis.

Methods

The authors propose two primary model-agnostic techniques to bolster OOD performance:

  1. Invariant Risk Minimization (IRM) for Covariate Shift: To handle variations in molecular size and scaffold complexity, the authors adapt IRM. The goal is to force the model to focus on "invariant" features—the specific chemical substructures that dictate a reaction center—rather than "spurious" features that might change as molecules get larger or more complex.
  2. Concept Enhancement via Energy-Based Models (EBM) for Label Shift: Retrosynthesis is a "many-to-many" problem; one product can often be made via multiple valid pathways. However, datasets usually only record one "ground truth" reaction. The authors use an EBM to transform binary labels into a continuous probabilistic space. This "denoises" the training data by identifying potentially feasible but unobserved reactions, helping the model generalize to novel reaction strategies.
Figure 1: Three train-test pairs of reactions exhibiting label shift OOD, regular ID, and covariate shift OOD (from left to right) were analyzed. In the case of label shift OOD, a distributional shift manifests in retro-strategies within similar target molecules. Conversely, in the case of covariate shift OOD, a distributional shift occurs in target molecules within similar retro-strategies.

Datasets

The study uses the standard USPTO-50K dataset but introduces two new OOD splits:

  • USPTO50K_T (Label Shift): Reactions are split based on the frequency and type of reaction templates. The OOD set contains templates that are rare or entirely unseen during training.
  • USPTO50K_M (Covariate Shift): Reactions are split based on the complexity of the target molecule, using molecular size and scaffold type as criteria. The OOD set contains the most complex molecules.
Figure 4: Molecular size and scaffold ID/OOD dataset split for covariate shift dataset USPTO50K_M.

Key Insights and Ideas

  • Disentangling the Shift: The authors argue that you cannot fix OOD performance without knowing what is shifting. Figure 1 illustrates how a shift can happen in the strategy (label) or the molecule itself (covariate).
  • The "Non-Mutual Exclusivity" of Templates: In image classification, a "Cat" is not a "Dog." In chemistry, a single reaction might be described by multiple valid templates depending on how much of the surrounding structure you include (the "radius"). The authors show that while "minimal templates" (radius=0) are mutually exclusive, "retro-templates" (radius=1+) are not, which complicates how models learn.
  • Template-Free Models Aren't a Silver Bullet: A common belief is that template-free models (which treat retrosynthesis like a translation task) generalize better to new reactions. The authors find this isn't necessarily true; these models still struggle significantly with unseen minimal templates.
Figure 2: Minimal-templates and retro-templates. Left: In the image classification task, cats and dogs are typically regarded as mutually exclusive, while ragdolls, bulldogs, and short-haired dogs are not. Though image D2 is only annotated as a bulldog, it has the potential label of a short-haired dog. Right: In retrosynthesis, similar to the image D2, both labels B2 and B3 are viable options for generating the correct reaction C2, but only one template B2 is exposed as the positive label in the training dataset due to non-mutual exclusivity.

Key Results

  • Performance Cliff: When moving from in-distribution (ID) to OOD test sets, all baseline models (GLN, MT, GraphRetro, etc.) saw a massive drop in Top-1 accuracy—often between 30% and 40%.
  • Mitigation Success: The proposed IRM and Concept Enhancement techniques provided consistent gains. For example, on the label shift benchmark, the enhanced models showed an average performance improvement of 4.6%.
  • Template Assembly: The authors discovered that the Molecular Transformer (MT) model is capable of "inventing" new reaction strategies by assembling components of different templates it saw during training, a key advantage for OOD generalization.

Limitations and Future Work

The authors acknowledge that their benchmarks are currently limited to the open-source USPTO dataset. Because many reaction databases are proprietary, the full extent of OOD challenges in industrial settings remains to be seen. Future work could involve extending these benchmarks to larger, private datasets and exploring more complex multi-step retrosynthesis scenarios where OOD errors can compound. Additionally, while IRM helped, the improvement was sometimes marginal due to "concept drift" in the datasets that IRM cannot yet fully account for.

Summary updated

Abstract

Machine learning-assisted retrosynthesis prediction models have been gaining widespread adoption, though their performances oftentimes degrade significantly when deployed in real-world applications embracing out-of-distribution (OOD) molecules or reactions. Despite steady progress on standard benchmarks, our understanding of existing retrosynthesis prediction models under the premise of distribution shifts remains stagnant. To this end, we first formally sort out two types of distribution shifts in retrosynthesis prediction and construct two groups of benchmark datasets. Next, through comprehensive experiments, we systematically compare state-of-the-art retrosynthesis prediction models on the two groups of benchmarks, revealing the limitations of previous in-distribution evaluation and re-examining the advantages of each model. More remarkably, we are motivated by the above empirical insights to propose two model-agnostic techniques that can improve the OOD generalization of arbitrary off-the-shelf retrosynthesis prediction algorithms. Our preliminary experiments show their high potential with an average performance improvement of 4.6%, and the established benchmarks serve as a foothold for further retrosynthesis prediction research towards OOD generalization.
BibTeX
@inproceedings{yu2024retroood,
  author = {Yemin Yu and Luotian Yuan and Ying Wei and Hanyu Gao and Xinhai Ye and Zhihua Wang and Fei Wu},
  title = {{RetroOOD: Understanding Out-of-Distribution Generalization in Retrosynthesis Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i1.27791},
}
Figure