Retrosynthesis planning — working backward from a target molecule to a sequence of chemical reactions that ends in cheap, commercially available building blocks — has recently become tractable for small, specialized language models. A 1.5-billion-parameter agent trained to orchestrate single-step reaction predictors now solves 94.21% of the hard Retro*-190 benchmark RetrOrchestrator, and a 7B agent trained with reinforcement learning reaches 55.79% Pass@1 on the same set and 73.30% out-of-distribution on ChEMBL-1000 Retro-R1. But every one of these systems reads molecules as text: a SMILES string, which is just a graph serialized into characters. The same molecule has combinatorially many equivalent SMILES — different atom traversal orders, branch placements, ring-closure numbering — and no one has checked whether the planning agents actually care. In code models, the analogous question has a brutal answer: renaming variables, which changes nothing about what a program does, flips model predictions in 90–99% of cases with as few as 1–3 token edits CodeAttack, and structural refactoring cuts code-translation Pass@1 by at least 17.97% CoTR.
We propose to import that adversarial lens into retrosynthesis, and then to close the gap it exposes. The core claim is two-part. First, compact retrosynthesis agents over-rely on serialization surface form — the accidental order in which a molecule's atoms happen to be written down — rather than on the molecular graph itself, and this reliance can be measured with a budgeted worst-case attack over each molecule's equivalence class of SMILES. Second, this fragility can be trained away by mixing verified, adversarially-mined surface-form variants into reinforcement-learning training, without sacrificing clean-input performance. The proposal treats adversarial vulnerability as the diagnostic, retrosynthetic search as the application regime, LLM-based data generation as the supply line for verified perturbations, and domain specialization as the model class under test — with search-space complexity playing a double role, since the attack itself is a combinatorial search over linearization space, mirroring the planner's search over reaction trees.
Why This Gap Is Open
The closest existing work stops one step short on each side. Auto-SPT: Automating Semantic Preserving Transformations for Code uses an LLM as a meta-programmer that designs semantics-preserving code transformations, validates them with unit tests, and chains them with beam search to find worst-case variants — raising worst-case embedding distance on CodeBERT from 0.243 to 0.947 on CodeContests — but it operates entirely on Python programs and trains encoders, not sequential decision-makers inside a search loop. Retrosynthesis Prediction Enhanced by In-Silico Reaction Data Augmentation (RetroWISE) augments chemical training data by generating new reactions from unpaired molecules, gaining 8.6% top-1 accuracy on USPTO-50K, but it generates new instances, never adversarial variants of existing ones, and never touches multi-step planners. ChemPRM: Improving Retrosynthesis by Structured Intermediate Process Reward shows surface form already matters benignly in this domain — its root-aligned SMILES (R-SMILES) expansion of 50,000 reactions into roughly 900,000 examples was central to its 58.5% top-1 / 86.9% top-10 USPTO-50K results — but studies only single-step disconnection and only random representation variation. Meanwhile the multi-step agents themselves assume clean, canonical inputs: both RetrOrchestrator and Retro-R1 explicitly leave perturbation robustness untested, and AOT* never probes whether non-canonical SMILES derail its retrieval or tree expansion. The four-way intersection — adversarial perturbation, multi-step search, LLM-generated augmentation, compact specialized models — is unoccupied in this corpus.
Two results motivate rather than constrain the design. Adversarial Robustness for Code found that adversarial training alone buys only a 0–7% robustness gain, and real resilience required changing the representation itself; we therefore test augmentation and a representation-level fix. And Re-evaluating Retrosynthesis Algorithms with Syntheseus showed that reported retrosynthesis numbers are inflated by evaluation hygiene — deduplication alone lifted GLN top-5 accuracy by 5.8%, and ~90% accuracy on USPTO-50K collapses to ~45% on the messier Pistachio set — which tells us that surface-form sensitivity is exactly the kind of artifact the field's evaluations are prone to hiding.
Research Program
The first build is a molecule-level transformation miner, the chemical counterpart of Auto-SPT's meta-programmer. A large language model proposes surface-form transformations — atom-order permutation strategies, branch reordering, ring-bond closure reassignment, alternative aromatic-notation forms, explicit-versus-implicit hydrogen writing — and implements each as an RDKit script. Every candidate is validated by an executable check: the transformed string must map to the identical molecule under InChI comparison, the chemical analog of Auto-SPT's unit-test validation. This gives a verified pool of semantics-preserving molecular transformations, plus a separately labeled "near-equivalence" tier for tautomers and resonance forms, which are chemically interchangeable in context but not graph-identical and are not normalized by any canonicalizer — we keep them out of the strict-equivalence attack and report them separately rather than blur the boundary.
The second build is the attack search itself. The number of valid SMILES for a molecule grows combinatorially with atom count, so finding the worst case is a search problem in the same sense that retrosynthesis is Navigating the Combinatorial Complexity of Retrosynthetic Planning — an unguided walk over a branching factor of 100 for 10 steps visits $10^{20}$ states. We run a budgeted beam search over the equivalence class, guided by the victim agent's own value estimates, to find the linearization that most damages planning:
$$\tilde{s} \;=\; \arg\max_{s' \in \mathcal{E}(s)} \big( J(\pi, s) - J(\pi, s') \big), \qquad \mathcal{E}(s) = \{\, s' : \mathrm{InChI}(s') = \mathrm{InChI}(s) \,\}$$
where $s$ is a target's SMILES string, $\mathcal{E}(s)$ its equivalence class, $\pi$ the planning agent, and $J(\pi, s)$ an indicator of whether the agent solves the target within a fixed iteration budget. Three perturbation levels are recorded per target: canonical (the status quo), average-case (the mean over $K$ random linearizations), and worst-case (the minimum under a fixed attack budget).
The third build is the defense. We retrain the compact orchestrator with Scaffold-Aware Group-Relative Policy Optimization, following the public verl-retro-agent implementation of RetrOrchestrator, on a curriculum in which the environment presents target molecules and intermediate states in non-canonical forms, including forms mined by the attack search. In parallel we add one representation-level change motivated by Bielik et al.'s negative result on augmentation-only defenses: a NORMALIZE action that lets the agent canonicalize a state before comparing tree nodes and invoking tools, so duplicate detection and stock matching operate on canonical graphs rather than raw strings.
Proposed Experiments
Measurement runs on Retro*-190, ChEMBL-1000, and PDB-600, with Pistachio Hard as a stress set, all evaluated through the standardized deduplication and stereochemistry protocol of Syntheseus. Two victim agents bracket the model classes: the 1.5B SA-GRPO orchestrator (reported 94.21% on Retro*-190, 68.83% on PDB-600, about 10 tool calls per route) and the 7B PPO agent of Retro-R1 (55.79% Pass@1 on Retro*-190, 73.30% on ChEMBL-1000). We also isolate where fragility lives by testing the underlying single-step engines separately, since Syntheseus showed single-step quality dominates multi-step outcomes. Metrics: solve-rate drop from canonical to perturbed inputs, extra iterations consumed, duplicate-detection failure rate inside the search tree, and tool-routing consistency (whether the orchestrator picks the same single-step model for the same molecule written two ways). Success for the defense is a worst-case drop under 5 points on Retro*-190 while clean performance stays within 2 points of the reproduced 94.21% baseline.
Baselines
The "do nothing" reference point is the unmodified RetrOrchestrator agent on canonical inputs, whose reported 94.21% success on Retro*-190 and 68.83% on PDB-600 we first reproduce from its public codebase with the same seven single-step tools, the same stock database, and the same iteration cap; this number is what any defense must not degrade. The strongest existing competitor is that same agent evaluated under our perturbation protocol — the question is not whether we beat it on clean inputs but whether the gap between its clean and worst-case numbers is large, and then whether our retrained agent closes that gap at equal clean performance, equal tool inventory, equal iteration budget, and equal tuning effort. A second reference is the 7B Retro-R1 agent, run under the identical protocol, to see whether fragility shrinks with scale — directly probing the ChemPRM observation that gains saturate between 3B and 7B parameters. Two further baselines define the defensibility frontier: an input-side canonicalization wrapper that rewrites every incoming SMILES to its RDKit canonical form before the agent sees it (the cheapest possible defense, detailed below), and a random-augmentation retraining run in which the curriculum uses uniformly sampled linearizations rather than attack-mined ones — this last ablation isolates the proposal's key ingredient, the worst-case mining, since everything else about training is held identical.
Why not just…?
Why not just canonicalize the inputs with RDKit? This is the most serious objection, and for pure atom-reordering we expect it to close most of the input-side gap — which is precisely why it is a baseline rather than an afterthought. But three channels survive input canonicalization. First, the agent writes SMILES itself when it calls single-step tools and compares tree nodes; its own emissions can be non-canonical, and duplicate detection or stock matching on non-canonical strings wastes search iterations — an agent-behavior problem no input filter fixes. Second, tool-returned candidates arrive in tool-specific orderings the planner cannot pre-canonicalize without paying extra calls, and RetrOrchestrator sits on the query-count/latency Pareto frontier precisely because it spends only about 10 calls per route. Third, tautomers and resonance forms are not normalized by any canonicalizer, and canonical output also varies across toolkit versions — the same class of quiet evaluation hazard Syntheseus exposed with deduplication and stereochemistry. We will report exactly what fraction of the total fragility canonicalization removes; if it removes essentially all of it, the honest conclusion is that the vulnerability is shallow at the input boundary, and the lasting contribution becomes the measurement protocol plus the agent-emission finding. Both outcomes are informative, and the experiments are designed so neither is a null result.
Why not just prompt a frontier LLM instead of training compact agents? The corpus answers this directly: an off-the-shelf LLM achieves 9.47% on Retro*-190 versus 94.21% for the specialized 1.5B orchestrator RetrOrchestrator, and embedding LLMs into step-wise search collapses hard-task solve rates to near 0% LLM-Syn-Planner. Even the strongest prompt-based system, AOT*, depends on GPT-4o-class models with retrieval, losing 20–40% of its solve rate when retrieval is removed. Robustness training only makes sense on a model class worth defending, and the compact agents are that class.
Why not just do random SMILES augmentation instead of worst-case mining? Random variation as augmentation is already established and effective — R-SMILES expansion underpins ChemPRM's gains, and RetroWISE's filtering pipeline added 8.6% top-1 on USPTO-50K — so this is a genuine threat to the proposal's added machinery. The counter-evidence is that in the code domain, defense required distance-maximizing selection, not random sampling: CoTR kept robust degradation at 0.67% only by filtering variants that maximized semantic distance, and Auto-SPT needed beam search rather than uniform sampling to find hard cases. The random-versus-mined ablation settles it empirically at matched compute and matched curriculum: if random augmentation matches worst-case augmentation on the worst-case evaluation, we adopt the cheaper recipe and report the attack results as the measurement contribution. We would rather be wrong about the mechanism than defend it against the data.
Decisive Checks
Three outcomes decide the proposal. The measurement phase succeeds if the canonical-to-worst-case solve-rate gap on Retro*-190 exceeds the noise floor of the Syntheseus protocol by a clear margin for at least one agent class — establishing that the chemistry domain inherits the natural-channel fragility documented for code. The defense phase succeeds only if the combined mined-augmentation and normalize-action retraining beats both the canonicalization wrapper and random augmentation on worst-case inputs while staying within 2 points of 94.21% clean. And a final scaling check retrains at 1.5B and 3B to test whether robustness, like accuracy in ChemPRM, comes from data and objectives rather than parameters — with a perturbation-averaged solve rate proposed as a standard reporting metric, so that future planners are judged not only on the clean, canonical inputs that current benchmarks happen to supply.
References
- RetrOrchestrator: A Multi-Step Retrosynthesis Agent Dynamically Orchestrating Single-Step Transition Models
- Retro-R1: LLM-based Agentic Retrosynthesis
- CodeAttack: Code-Based Adversarial Attacks for Pre-trained Programming Language Models
- Assessing and improving syntactic adversarial robustness of pre-trained models for code translation
- Auto-SPT: Automating Semantic Preserving Transformations for Code
- Retrosynthesis prediction enhanced by in-silico reaction data augmentation
- ChemPRM: Improving Retrosynthesis by Structured Intermediate Process Reward
- AOT*: Efficient Synthesis Planning via LLM-Empowered AND-OR Tree Search
- Adversarial Robustness for Code
- Re-evaluating Retrosynthesis Algorithms with Syntheseus
- Towards "AlphaChem": Chemical Synthesis Planning with Tree Search and Deep Neural Network Policies
- LLM-Augmented Chemical Synthesis and Design Decision Programs