TL;DR
RetroOOD provides a rigorous framework for understanding why retrosynthesis models fail when they encounter new molecules or reaction types. By disentangling covariate shift (changes in molecular structure) from label shift (changes in reaction strategies) and introducing new OOD benchmarks, the authors propose model-agnostic techniques that improve generalization by an average of 4.6%.
What Problem Does This Paper Address?
Retrosynthesis—the process of identifying the simpler building blocks needed to synthesize a complex molecule—is a cornerstone of organic chemistry. While machine learning models have become quite good at this on standard benchmarks, they often "break" when deployed in the real world.
The core issue is Out-of-Distribution (OOD) generalization. Most models are trained and tested on data that look similar (i.e., they are independent and identically distributed, or i.i.d.). However, a model trained on simple molecules might fail on complex ones (covariate shift), or a model might struggle to predict a reaction type it hasn't seen frequently, even if the target molecule is familiar (label shift). Before this paper, there was no systematic way to measure or mitigate these specific types of shifts in retrosynthesis.
Methods
The authors propose two primary model-agnostic techniques to bolster OOD performance:
- Invariant Risk Minimization (IRM) for Covariate Shift: To handle variations in molecular size and scaffold complexity, the authors adapt IRM. The goal is to force the model to focus on "invariant" features—the specific chemical substructures that dictate a reaction center—rather than "spurious" features that might change as molecules get larger or more complex.
- Concept Enhancement via Energy-Based Models (EBM) for Label Shift: Retrosynthesis is a "many-to-many" problem; one product can often be made via multiple valid pathways. However, datasets usually only record one "ground truth" reaction. The authors use an EBM to transform binary labels into a continuous probabilistic space. This "denoises" the training data by identifying potentially feasible but unobserved reactions, helping the model generalize to novel reaction strategies.
Datasets
The study uses the standard USPTO-50K dataset but introduces two new OOD splits:
- USPTO50K_T (Label Shift): Reactions are split based on the frequency and type of reaction templates. The OOD set contains templates that are rare or entirely unseen during training.
- USPTO50K_M (Covariate Shift): Reactions are split based on the complexity of the target molecule, using molecular size and scaffold type as criteria. The OOD set contains the most complex molecules.
Key Insights and Ideas
- Disentangling the Shift: The authors argue that you cannot fix OOD performance without knowing what is shifting. Figure 1 illustrates how a shift can happen in the strategy (label) or the molecule itself (covariate).
- The "Non-Mutual Exclusivity" of Templates: In image classification, a "Cat" is not a "Dog." In chemistry, a single reaction might be described by multiple valid templates depending on how much of the surrounding structure you include (the "radius"). The authors show that while "minimal templates" (radius=0) are mutually exclusive, "retro-templates" (radius=1+) are not, which complicates how models learn.
- Template-Free Models Aren't a Silver Bullet: A common belief is that template-free models (which treat retrosynthesis like a translation task) generalize better to new reactions. The authors find this isn't necessarily true; these models still struggle significantly with unseen minimal templates.
Key Results
- Performance Cliff: When moving from in-distribution (ID) to OOD test sets, all baseline models (GLN, MT, GraphRetro, etc.) saw a massive drop in Top-1 accuracy—often between 30% and 40%.
- Mitigation Success: The proposed IRM and Concept Enhancement techniques provided consistent gains. For example, on the label shift benchmark, the enhanced models showed an average performance improvement of 4.6%.
- Template Assembly: The authors discovered that the Molecular Transformer (MT) model is capable of "inventing" new reaction strategies by assembling components of different templates it saw during training, a key advantage for OOD generalization.
Limitations and Future Work
The authors acknowledge that their benchmarks are currently limited to the open-source USPTO dataset. Because many reaction databases are proprietary, the full extent of OOD challenges in industrial settings remains to be seen. Future work could involve extending these benchmarks to larger, private datasets and exploring more complex multi-step retrosynthesis scenarios where OOD errors can compound. Additionally, while IRM helped, the improvement was sometimes marginal due to "concept drift" in the datasets that IRM cannot yet fully account for.
Summary updated