Language models asked to explain how a chemical reaction happens, step by step, break down once the explanation runs past a handful of steps. They also tend to propose chemically impossible intermediate molecules even when they get the final product right. The usual reading is that this is a reasoning ceiling: errors pile up as the chain gets longer. There is a second reading that nobody has tested. Each intermediate the model writes is a molecule spelled out as a text string, and models are known to misread molecules spelled in unfamiliar ways. The model may simply be writing intermediates in idiosyncratic spellings and then misreading its own handwriting. This proposal separates the two explanations by measurement and asks which one dominates.
The test bed is light-driven radical chemistry, and in particular proton-coupled electron transfer (PCET), in which a molecule loses an electron and a proton together so that it avoids a high-energy charged state. These reactions suit the question for three reasons. Their intermediates carry unpaired electrons, which are rare and awkward to write in standard notation. Whether a reaction succeeds depends on picking out one exact carbon atom along a chain. And no current mechanism benchmark covers them. If notation drift exists, it should be largest here, and here it can also be checked by program.
Core Claim
Failure on multi-step mechanisms can be split into two parts. The first is per-step chemical error. The second is representation drift: extra error that appears only because the model reads intermediates in the string form it wrote itself, rather than in a clean, standardized form. Let $p_i$ be the model's accuracy on step $i$ when it is handed the correct previous intermediate in canonical form, and let $S_k$ be its success rate on the full $k$-step chain generated on its own. We define the drift excess as
$$D_k = \prod_{i=1}^{k} p_i \;-\; S_k ,$$
which is zero if step errors are independent and the model's own intermediates are as readable as gold ones. The product form follows from treating steps as independent given correct inputs. A second chained condition rewrites every model-generated intermediate into canonical form before the next step is posed, using RDKit, the standard open-source chemistry toolkit, which can parse a molecule string and re-spell it in a fixed order. We call this condition re-canonicalization. Its success rate is $S_k^{\text{canon}}$.
The hypothesis makes four predictions. $D_k$ is positive and grows with $k$. Re-canonicalization recovers a substantial share of it. The share is larger for radical mechanisms than for the polar textbook mechanisms in existing benchmarks. Giving atoms stable index labels across steps closes most of the remaining error on site selection, meaning which carbon gets attacked. If $D_k$ is near zero, the step wall is a genuine reasoning limit, and that negative answer is also worth knowing.
Relation to Prior Work
Two lines of work motivate the question.
The first line found surface-form fragility in chemistry, but only on static, single-step tasks. ChemDIRT varies prompt phrasing and molecular encoding (SMILES, IUPAC names, images) and reports Llama 3 accuracy on ClinTox ranging from 9.5% to 90.5% across equivalent prompts. Its "mechanistic" category is basic, and it does not chain steps. MolecularIQ uses symbolically verified graph questions and shows two things. Models lean on canonical SMILES and fail on randomized or Kekulized spellings. Their accuracy also drops 5–30% when asked for exact atom indices rather than counts. Both findings concern isolated molecules. We carry the same perturbations into chains where the model produces the strings itself.
The second line found the step limit without asking about notation. oMeBench documents the ceiling past five elementary steps and the hallucinated intermediates, uses only single-string SMILES, and explicitly excludes photoredox and radical chemistry. We reuse its alignment metric, oMeS. We also use its oMe-Gold set as the polar control.
Several papers supply the method.
- ChemPRM supervises structured intermediate stages and trains on randomized-SMILES augmentation. It stays within single-step heterolytic disconnections, meaning bonds that break with both electrons going to one fragment. Its augmentation is a baseline we must beat.
- Synthegy has a rule engine enumerate legal electron-pushing moves and an LLM rank them. Its engine generates only closed-shell polar moves, so we extend the move set to radical steps.
- When Splitting Makes Stronger argues that isolating subtasks in clean contexts bounds error propagation. It is the strongest rival explanation for any gain from our per-step protocol. That is why we separate "clean context" from "clean spelling."
One paper constrains the design. How often are errors in natural language reasoning due to paraphrastic variability? finds that task fine-tuning can increase sensitivity to wording. So we cannot assume a fine-tuned model is notation-robust; we measure it. We adopt that paper's paraphrastic-consistency idea for our invariance metric. FormInv shows that 3.1% of faulty paraphrase groups can reorder model rankings. So every perturbed molecule string we generate is round-trip verified to the same InChIKey (a standard molecular fingerprint) before use.
Research Program
The benchmark. We first build RadMech, a mechanism set of light-driven radical reactions. Seed reactions come from the chemistry literature in this file, with hand-verified elementary steps:
- imidazolium N–S bond cleavage releasing SO₂F radicals (Li et al.) or CF₂H/CF₂D radicals (Sun et al.);
- amine–sulfonium electron donor–acceptor (EDA) complexes, pairs of molecules that absorb light together (Wang et al.);
- cyanomethylation followed by cyclization (Gao et al.);
- photo/cobalt dual catalysis ( A photo- and cobalt-catalyzed highly selective and divergent hydrofunctionalization of 1,3-dienes with phenols);
- PCET generation of alkoxy radicals followed by 1,5-hydrogen-atom transfer from the δ-carbon (Kachore et al.).
Each step is typed: excitation, single-electron transfer, concerted PCET, homolysis, hydrogen-atom transfer (HAT), radical addition, or radical-polar crossover.
The seeds are expanded by R-group substitution in the manner of oMe-Silver. The 1,5-HAT site is not taken from an LLM proposal. It is computed from graph topology, as the carbon five bonds from the oxygen. Geometric feasibility is checked by generating 3D conformers and testing the 2.5–2.7 Å donor–acceptor window. Every answer about site selection is therefore verified by program, with no annotator needed.
Five renderings. Each intermediate is rendered in five surface forms: canonical SMILES; randomized SMILES; Kekulized SMILES; atom-mapped SMILES, where every atom keeps a fixed numeric label across all steps; and an IUPAC-style name where one can be generated. Radical centers use the bracket notation RDKit round-trips.
Five protocols. We run every model under five protocols:
- single-pass full mechanism;
- chained self-generated steps;
- chained steps with re-canonicalization;
- chained steps with atom-mapped re-rendering;
- isolated steps with gold previous intermediates.
The isolated-step protocol gives $p_i$, and the chained protocols give $S_k$ and $S_k^{\text{canon}}$. Running isolated steps in a fresh context, versus appending them to the running transcript, separates context isolation from spelling cleanup.
Polar control. The same protocols run on oMe-Gold. This tells us whether drift is specific to radicals or general to mechanisms.
Baselines
Naive reference. A frontier generalist model prompted once with ordinary chain-of-thought (step-by-step reasoning written out in text) to produce the whole mechanism, scored with oMeS and site accuracy. This is the "do nothing" point. Any method that cannot beat it on RadMech is not worth its cost. Its weakness is already reported: MolecularIQ finds that failed reasoning traces are longer than successful ones.
Fine-tuned mechanism model. The oMeBench recipe: supervised fine-tuning of the same 4B Qwen-3 base on oMe-Silver, plus a RadMech-templated training split of matched size. oMeBench reports a "50% improvement" of Qwen-3-SFT over Claude-Sonnet-4 and GPT-4o on its benchmark. The file gives no absolute oMeS value, so the target is set by our own reproduction on oMe-Gold.
Augmentation-trained model. The same fine-tuning plus ChemPRM-style randomized-SMILES augmentation. ChemPRM reports 58.5% Top-1 and 86.9% Top-10 on USPTO-50k, but that is single-step retrosynthesis and not comparable. No reported result was found on Lacuna for any model on radical or PCET mechanism elucidation.
Divide-and-conquer prompting. Isolated-context step prompting following When Splitting Makes Stronger, without re-canonicalization. This is the ablation that isolates our key ingredient. Gains from clean context alone show up here; gains from clean spelling show up only in the next step up.
Enumerator-plus-judge. A Synthegy-style system in which our extended radical move enumerator proposes legal next steps and the LLM ranks them. The model never writes strings, so drift is impossible by construction.
Fairness. All prompted conditions use identical model checkpoints, the same decoding settings, and the same per-mechanism token budget. All fine-tuned conditions share base model, data volume, and a fixed hyperparameter sweep. Beating a baseline means a higher $S_k$ at $k > 5$, with bootstrap confidence intervals that do not overlap, on held-out radical scaffolds.
Why not just…?
Why not just prompt a frontier model with chain-of-thought? Because the step wall in oMeBench was measured on exactly such models, and polar textbook chemistry is easier than radical chemistry. This baseline is run anyway. The interesting output is not whether it fails but how its failure splits between $p_i$ and $D_k$.
Why not just train on randomized SMILES so the model stops caring about spelling? This is the real threat. If augmentation drives $D_k$ to near zero, the cheap fix suffices and our tool-based re-canonicalization is unnecessary. The experiment settles it directly, because $D_k$ is measured for the augmentation-trained model. One caution: the paraphrase paper ( How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?) suggests fine-tuning can add brittleness elsewhere, so we also check its consistency on held-out renderings, such as IUPAC names, that were not in the augmentation.
Why not just split the problem into steps, which is known to help? Splitting changes two things at once: context length and the spelling of what is passed forward. The divide-and-conquer arm keeps the model's own strings. The re-canonicalization arm cleans them. Only the difference between the two is attributed to notation.
Why not avoid strings entirely with an enumerator and judge? If the Synthegy-style system dominates, the practical conclusion is that models should rank moves rather than write intermediates, which is a clear and useful answer. It does require a hand-built radical move enumerator, and its coverage limits on mixed radical-polar steps are themselves a measurement.
Proposed Experiments
Decomposition. The main experiment computes $p_i$, $S_k$, $S_k^{\text{canon}}$ and $D_k$ for steps 1 through 8 and beyond, for every model and both chemistry families. We report what fraction of $D_k$ re-canonicalization recovers.
Decisive check 1: does drift exist? We test whether $D_k$ is significantly positive past five steps on RadMech. A permutation test shuffles step outcomes across mechanisms to estimate the independence baseline.
Decisive check 2: is it specific to radicals? We test whether the recovered fraction is larger on radical mechanisms than on oMe-Gold, controlling for chain length.
Decisive check 3: do stable atom labels fix site selection? We test whether the atom-mapped protocol raises δ-site accuracy on the 1,5-HAT subset more than it raises overall step accuracy. That pattern would tie the MolecularIQ indexing gap to mechanism failure.
Secondary analyses. Invariance is reported as agreement across the five renderings of the same step. Error analysis classifies each failed chain as chemistry-wrong (the intermediate is valid but incorrect) or parse-wrong (the next step misreads a valid intermediate). The decision rule: if all three checks come out negative, we report that the step wall is a reasoning limit that survives clean notation, and the work becomes a radical-mechanism benchmark with that negative result attached.
References
- ChemDIRT: A Diversified Instruction, Representation, and Task Benchmark for Robust Chemistry-LLM Evaluation
- MolecularIQ: Characterizing Chemical Reasoning Capabilities Through Symbolic Verification on Molecular Graphs
- oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning
- ChemPRM: Improving Retrosynthesis by Structured Intermediate Process Reward
- Chemical reasoning in LLMs unlocks steerable synthesis planning and reaction mechanism elucidation
- When Splitting Makes Stronger: A Theoretical and Empirical Analysis of Divide-and-Conquer Prompting in LLMs
- How Often Are Errors in Natural Language Reasoning Due to Paraphrastic Variability?
- FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
- A practical fluorosulfonylating platform via photocatalytic imidazolium-based SO2F radical reagent
- A practical photocatalytic strategy for radical (deuterio)difluoromethylation using imidazolium reagents
- Controllable tertiary amine-promoted photoactivation metal-free carbonylation of aryl sulfonium salts to aryl carboxylic acid derivatives
- Visible-light induced tandem radical cyanomethylation and cyclization of N-aryl acrylamides: access to cyanomethylated oxindoles
- A photo- and cobalt-catalyzed highly selective and divergent hydrofunctionalization of 1,3-dienes with phenols
- Direct site-selective C(sp<sup>3</sup>)–H functionalization of unprotected non-activated alcohols
- How often are errors in natural language reasoning due to paraphrastic variability?