Beyond the Training Domain: Robust Generative Transition State Models for Unseen Chemistry

Samir Darouich, Jacob Toney, Weiliang Luo, Johannes Kästner, Mathias Niepert, Heather J. KulikarXiv 2026 · DOI · arXiv

TL;DR

Generative AI models for predicting chemical transition states often fail when they encounter elements outside their narrow training set (usually small organic molecules). This paper introduces a self-supervised pretraining strategy using "pseudo-reactions" derived from easy-to-calculate equilibrium structures, slashing data requirements by 75% and enabling accurate predictions for complex transition-metal chemistry.

What Problem Does This Paper Address?

In computational chemistry, the Transition State (TS) is the "mountain pass" a reaction must cross to turn reactants into products. Finding these states is notoriously difficult and expensive, often requiring thousands of iterations of high-level quantum mechanical calculations (like Density Functional Theory, or DFT).

While recent generative machine learning models can predict these structures in seconds, they have a "blind spot." Most are trained on datasets like Transition1x, which consist almost entirely of small organic molecules (Hydrogen, Carbon, Nitrogen, Oxygen). When these models encounter "unseen" chemistry—like a transition metal catalyst or a silicon atom—they fail spectacularly, often "hallucinating" unphysical geometries where atoms overlap or bonds are impossibly short. The authors set out to bridge this gap: how do we make these models robust enough to handle the entire periodic table without needing millions of new, expensive TS examples?

Methods

The researchers focused on state-of-the-art generative models, specifically React-OT (which uses optimal transport and flow matching) and AEFM (Adaptive Equilibrium Flow Matching). Their core innovation is a self-supervised pretraining strategy:

  1. Pseudo-Reactions: Instead of needing real (and expensive) reaction paths, they use equilibrium conformers—different spatial arrangements of a molecule at rest.
  2. The Ranking Trick: For a given molecule, they rank three conformers by energy. They treat the highest-energy one as a "pseudo-TS," the second-highest as a "pseudo-reactant," and the lowest as a "pseudo-product."
  3. Exposure Therapy: By training the model on these pseudo-reactions first, the model learns the "rules of geometry" for new elements (like bond lengths and angles) before it ever tries to predict a real, complex reaction.

Datasets

The study introduces two major new benchmarks to test chemical "novelty":

  • Transition1x-2p3p4p: A dataset of ~14,000 reactions where atoms in organic molecules are swapped for elements in lower periods of the periodic table (e.g., swapping Carbon for Silicon or Germanium).
  • Transition1x-TMC: A massive dataset of over 36,000 reactions involving ten different Transition Metal Complexes (including Pt, Rh, and Pd), representing a significant leap in complexity from standard organic sets.
  • Transition1x-OOD: A smaller "out-of-distribution" set featuring more exotic substitutions like Boron-Nitrogen bonds.

All structures were computed at the GFN2-xTB semi-empirical level, with subsets validated using high-level DFT.

Key Insights and Ideas

  • The "Organic Bias": Models trained only on HCNO (Hydrogen, Carbon, Nitrogen, Oxygen) try to force organic bond lengths onto everything. For example, they might predict a Metal-Carbon bond using the length of a Carbon-Carbon bond, leading to massive energetic errors.
  • Geometry over Mechanism: You don't need a real reaction to learn what a reasonable atomic environment looks like. Pretraining on "fake" reactions (pseudo-reactions) provides enough structural context to stabilize the model for real tasks.
  • Data as a Bottleneck: High-quality TS data is the scarcest resource in the field. This method turns abundant, cheap equilibrium data into a "force multiplier" for the scarce TS data.

Key Results

  • Dramatically Lower Error: On the Transition-Metal (TMC) benchmark, the pretraining strategy dropped the median structural error (RMSD) from 0.39 Å to 0.19 Å.
  • Energetic Realism: For the elemental swap dataset, the median energetic error plummeted from a nonsensical 197 kcal/mol (vanilla model) to a much more reasonable 31 kcal/mol.
  • Extreme Data Efficiency: Pretrained models achieved the same accuracy as "vanilla" models using only 25% of the fine-tuning data. In ultra-low data regimes (just 50 real reactions), the model could still successfully predict complex platinum-catalyzed methane activation.
  • DFT Transferability: Models trained on faster, lower-level calculations (xTB) provided such good "guesses" that 75-82% of them successfully converged to the correct structure when refined with expensive DFT.

Limitations and Future Work

  • Ligand Diversity: While the model handles new metals well, the authors note that generalizing to entirely new types of organic ligands (the molecules attached to the metals) remains a challenge.
  • Active Learning: The authors suggest that future work could use "uncertainty-aware" active learning to automatically identify which pseudo-reactions would be most helpful for the model to learn next.
  • Beyond Semi-Empirics: While xTB is a great compromise, the ultimate goal is a model that can produce DFT-quality results directly across the entire periodic table without any secondary refinement.

Summary updated

Abstract

Transition states (TSs) govern the rates and outcomes of chemical reactions, making their accurate prediction a central challenge in computational chemistry. Although recent machine-learning models achieve near chemical accuracy in the prediction of TS structures and the associated reaction barriers for small organic reactions, their ability to generalize beyond the training domain remains largely unexplored. Here, we introduce targeted benchmarks to probe chemical and structural novelty in generative TS prediction. Building on Transition1x, a large-scale dataset of reactions involving small organic molecules, we construct curated extensions incorporating controlled elemental substitutions and diverse transition-metal complexes (TMC). These benchmarks reveal fundamental limitations of generative models in the generalization to previously unseen elements. As a result, they produce unphysical geometries and large energetic errors, even for reactions structurally similar to well-predicted organic systems. To address this challenge, we introduce a self-supervised pretraining strategy based on equilibrium conformers that exposes generative TS models to novel chemical environments prior to targeted fine-tuning. Across the newly proposed benchmarks, self-supervised pretraining substantially improves TS prediction for previously unseen systems, lowering the median RMSD of TS geometries on T1x-TMC reactions from 0.39 to 0.19 $\mathring{A}$ and reducing fine-tuning data requirements by up to 75%, enabling reliable performance even in low-data regimes. Overall, the integration of generative TS models with self-supervised pseudo-reaction pretraining provides an efficient, scalable, and chemically robust framework for elucidating TSs well beyond the small organic molecule domain, establishing a foundation for investigating complex and catalytically relevant reaction landscapes.
BibTeX
@misc{darouich2026beyond,
  author = {Samir Darouich and Jacob Toney and Weiliang Luo and Johannes Kästner and Mathias Niepert and Heather J. Kulik},
  title = {{Beyond the Training Domain: Robust Generative Transition State Models for Unseen Chemistry}},
  year = {2026},
  eprint = {2601.16469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.16469},
}
Figure