Mechanism or Matching: A Shortcut Audit of Multimodal Synthesis Planning

Generative models that design molecules under synthesizability constraints — that is, models that output a recipe of real reactions over purchasable starting materials rather than a bare molecular graph — have started to lean on natural-language and visual context to make better decisions. RetroInText predicts retrosynthetic precursors while fusing 2D graphs, 3D geometry, and LLM-written reaction narratives (RetroInText); Chemma-RC recommends catalysts and solvents using molecular graphs plus retrieved literature text (Chemma-RC). Meanwhile, an adjacent literature on the multimodal reasoning gap has documented an uncomfortable fact: models that appear to reason over images and text often exploit superficial textual cues instead. When chart titles containing answer hints were replaced with misleading text, accuracy collapsed from as high as 79% to the 30% range (Plot Twist). Nobody has checked whether chemistry's multimodal gains are real reasoning or the same trick.

This matters for more than accounting. Synthesizability-constrained generators are increasingly guided by learned oracles — condition recommenders, feasibility scorers, route rankers. If those oracles succeed by matching reagent names to substrate co-occurrence statistics rather than by reading chemical structure, they will silently endorse routes that fail in the flask, which is precisely the "execution gap" between algorithmic and practical synthesizability that Gao and Coley identify as the field's core failure mode. This proposal builds a perturbation-based audit for multimodal reaction-planning models, quantifies how much of each reported context gain survives adversarial controls, and then trains a shortcut-robust replacement whose use of context is verifiably structural.

Core Claim

The claim is that the reported benefits of textual and visual context in synthesis-planning models are currently unvalidated, that a Chart101-style adversarial probe suite can separate mechanistic grounding from linguistic matching, and that contrastive training against distractor context can keep the genuine signal while eliminating the shortcut. The evidence that motivates this is already in the literature but has never been connected. RetroInText's own ablation shows that removing its entire textual apparatus costs only 1.0 point of top-1 accuracy on RetroBench, dropping from 41.2% to 40.2% (RetroInText) — a suspiciously small return for a tripartite multimodal encoder, and exactly the pattern Plot Twist found before probing revealed that aggregate scores mask severe per-item fragility (Plot Twist). Chemma-RC, for its part, attributes a 7–17% improvement over prior baselines to retrieved literature text (Chemma-RC) — a much larger stake that has never been stress-tested. The multimodal reasoning literature supplies both the diagnosis and the method: error attribution on MATH-Vision shows 42.2% of failures are pure reasoning errors despite accurate perception (MATH-Vision), and GeoSQA showed that handing models clean text translations of diagrams does not rescue logical failure (GeoSQA), which together rule out the hope that better perception alone will fix this.

Research Program

The program has three phases: audit, remediate, and propagate.

The audit phase builds ChemAudit, a probe suite layered on existing benchmarks rather than a new dataset from scratch. It has three probe families. First, distractor-rationale probes: for each item on RetroBench (pathways for 128,469 molecules at depths of 2 to 13 steps) and USPTO-Condition/USPTO_500MT, we keep the molecular inputs intact and replace the true textual context with a plausible-but-wrong narrative — an LLM-generated description attributing, say, a palladium-catalyzed coupling to a nucleophilic substitution mechanism, or generic text about the molecule class that never mentions the actual transformation. This mirrors Plot Twist's misleading-title design, where the visual content was intact and only the superficial cue was corrupted (Plot Twist). A model that reads structure should be unaffected; a model that matches text should flip. We summarize this with a shortcut-susceptibility score,

$$\mathrm{SS} \;=\; \frac{A_{\mathrm{clean}} - A_{\mathrm{distractor}}}{A_{\mathrm{clean}}},$$

where $A$ is top-1 accuracy on the same items under clean versus distractor context, so $\mathrm{SS}=0$ means full resistance and $\mathrm{SS}=1$ means total collapse. Second, elementary structural probes: before asking which conditions a reaction needs, we ask which bond forms and which functional group transforms, using the atom-mapped reaction roles already present in USPTO data. This is the chemistry analogue of Chart101's axis-extrema probes; a model that cannot identify the reacting center yet recommends conditions confidently is matching reagent–substrate co-occurrence. We also report a collective-accuracy analogue — the fraction of items where the model answers every elementary probe and the main question correctly — since Plot Twist found this stays below 25% even for top chart models (Plot Twist). Third, layout and index probes for the visual pipeline: RxnScribe's F1 falls from 91.0% on single-line diagrams to 65.9% on graph-structured ones (RxnScribe), and RxnCaption pre-draws numbered boxes onto images before parsing (RxnCaption) — so we permute diagram layouts and shuffle the numbering to test whether these parsers reason over chemistry or over positional priors.

The remediation phase retrains the audited models with perturbation-aware objectives. The key ingredient is contrastive: for each item the model sees the true context and a matched distractor, and is trained — reusing the preference-learning machinery Chemma-RC already deploys in its REF ranking stage over high-throughput yield data (Chemma-RC) — to prefer structure-consistent outputs under both, so that only structural evidence can decide. We additionally force an auxiliary head to predict reacting centers before conditions, making structural reading a prerequisite rather than an option. This follows the precedent that decomposed, tool-augmented rationales lifted ChartQA human-written accuracy from 38.2% to 53.9% (Synthesize Step-by-Step): the fix is to make the reasoning step explicit and checkable, not to add capacity.

The propagation phase asks whether the audit matters downstream. We plug the raw and the remediated condition recommenders into a reaction-space generator in the style of SynFlowNet, which builds molecules through real reaction templates and achieved a 62% retrosynthesis solve rate with AiZynthFinder versus 0% for unconstrained fragment-based generation (SynFlowNet). The oracles re-rank sampled routes; we measure how often the shortcut-prone and robust oracles disagree, and whether routes endorsed only by the shortcut-prone oracle fail condition-level checks on held-out USPTO reactions.

Baselines

The naive reference point is a frontier generalist model prompted zero-shot with chain-of-thought. This is not a strawman: Chemma-RC itself reports that GPT-4o achieves 1.0% top-1 accuracy on the USPTO_500MT zero-shot condition task, against its own 25.9% (Chemma-RC), and Gemini-2.5-Pro trails the specialized RxnCaption-VL by 19.4 percentage points in HybridMatch F1 on RxnCaption-15k (RxnCaption). We run the same frontier models through ChemAudit to establish the floor and to test whether scale alone confers shortcut resistance; the expectation, grounded in MATH-Vision's finding that chain-of-thought can actively degrade multimodal accuracy (MATH-Vision), is that it does not.

The strongest competitors are the audited systems themselves, run from author checkpoints where available and otherwise retrained on their reported data. For retrosynthesis this is RetroInText, whose 41.2% top-1 accuracy on RetroBench (RetroInText) is the number to match; for condition recommendation it is Chemma-RC at 25.9% top-1 on USPTO_500MT (Chemma-RC); for diagram parsing it is the RxnCaption-VL pipeline on RxnCaption-15k and RxnScribe's 80.0% overall soft-match F1 on its 1,378-diagram ACS benchmark (RxnScribe). Fairness means identical items, identical perturbations, identical evaluation scripts, and a matched tuning budget: the remediated model gets no more compute or data than the baseline it replaces. Beating the baseline is defined two-sided — clean accuracy within noise of the reported number (at least 41.2% or 25.9% respectively on the clean splits), together with a distractor-induced flip rate at least halved relative to the original model and a substantially closed gap between probe-conditional and probe-unconditional accuracy. A remediated model that resists perturbation but loses clean accuracy has failed, and we will report it as such.

The ablation that isolates our key ingredient is a three-way comparison on the same items: the full model with true context, the text-removal ablation (for RetroInText, its own reported 40.2% (RetroInText) serves as the reference point), and the distractor condition. Removal and corruption answer different questions — removal measures how much text helps on average, corruption measures whether the help is structural — and the divergence between them is the paper's central measurement. A fourth arm, generic class-level text with no transformation-specific content, tests whether literature provenance matters at all, since RetroInText's narratives were generated offline by ChatGPT rather than retrieved from papers (RetroInText).

Why not just…?

Why not just drop the text and use the structure-only model? RetroInText's ablation already says text adds only one point, so the cheap answer is to delete it. But removal conflates shortcut with signal: a one-point average gain can hide a large shortcut on frequent reaction classes offset by nothing elsewhere, and Plot Twist demonstrated that aggregate accuracy is a poor guide to fragility — models looked fine on ChartQA until misleading titles cut them from 79% to the 30% range (Plot Twist). The stakes are also much larger for Chemma-RC, which credits literature text with up to 17 points (Chemma-RC). If genuine mechanistic signal exists, deleting it wastes it; our audit determines which is the case, and we are honest that "delete the text everywhere" is a possible and publishable outcome.

Why not just prompt GPT-4o or Gemini with chain-of-thought? The file's numbers are blunt: 1.0% top-1 on USPTO_500MT for GPT-4o (Chemma-RC), a 19.4-point F1 deficit for Gemini-2.5-Pro against a specialized parser (RxnCaption), and chain-of-thought that sometimes degrades multimodal accuracy rather than helping it (MATH-Vision). Frontier models are also, if anything, the most shortcut-exposed, since web-scale pretraining maximizes exactly the textual co-occurrence statistics we are probing for. We still run them, as the naive reference point above.

Why not just linearize everything — OCR the diagrams into SMILES and reason in pure text? GeoSQA already showed that handing models exhaustive natural-language descriptions of diagrams does not rescue logical failure (GeoSQA), and cascaded extraction pipelines compound early errors: RxnScribe cannot read molecular content itself and leans on external structure and OCR decoders, while degrading sharply on graph-layout diagrams (RxnScribe). Our index-permutation and layout probes directly measure whether the current visual pipelines are doing chemistry or reading positions, which a text-only pipeline cannot even ask.

Why not just train on more literature data? More of the same data entrenches the same shortcuts: the chart models Plot Twist probed were trained on ChartQA itself and still collapsed under title removal (Plot Twist). The targeted alternative — contrastive training against matched distractors — is the remediation we test, and it reuses machinery (preference learning over ranked candidates) that Chemma-RC has already validated for yield-based ranking (Chemma-RC), so it is not a speculative architectural bet.

Proposed Experiments and Decisive Checks

Experiment 1 constructs ChemAudit and runs the three probe families over RetroBench, USPTO-Condition and USPTO_500MT, and RxnCaption-15k, reporting $\mathrm{SS}$, flip rates, probe-conditional accuracy, and collective accuracy for RetroInText, Chemma-RC, the RxnCaption-VL pipeline, RxnScribe, and a frontier reference model. The decisive readout is asymmetric and we commit to both branches: a large $\mathrm{SS}$ for Chemma-RC's literature-text component would establish that a state-of-the-art condition recommender rides on textual matching, while $\mathrm{SS}$ near zero would validate the multimodal claim and partially refute our premise — also a finding, since it would be the first controlled evidence of that kind.

Experiment 2 runs the remediation on Chemma-RC and RetroInText with matched compute and data, evaluating clean and perturbed splits. Success is clean top-1 within noise of 25.9% (USPTO_500MT) and 41.2% (RetroBench) with at least half the distractor flip rate and a materially narrowed probe-conditional gap. As an external sanity check on whether remediation improves real chemical judgment rather than benchmark hygiene, we replicate the style of Chemma-RC's experimental ligand-selection validation (93.7% accuracy identifying XPhos in C-H arylation (Chemma-RC)) on held-out transformations; no comparable wet-lab result is reported on Lacuna for the other systems, so held-out USPTO splits are our substitute.

Experiment 3 propagates to generation: raw and remediated oracles re-rank routes from a SynFlowNet-style reaction-space generator, with the generator's own 62% AiZynthFinder solve rate (SynFlowNet) as the reference for route quality. We measure oracle disagreement on sampled routes and the condition-level failure rate of routes endorsed only by the shortcut-prone oracle. If the two oracles agree everywhere, the downstream stake is smaller than we argue and we will say so; if they disagree on a nontrivial fraction, the audit acquires a concrete cost: molecules whose recipes look fine to a matching-based model and are not.