Do Reaction Architectures Travel? A Tabular Benchmark and Zero-Cost Proxies for Chemical Seq2Seq Models

Chemists and machine-learning researchers have converged on a useful fiction: a chemical reaction is a sentence, and predicting a reaction is translating between two languages. Molecules are written as SMILES strings — one-dimensional token sequences where atoms, ring closures, and branch markers play the role of words — so "reactants → product" becomes a sequence-to-sequence translation problem. The Molecular Transformer established that a standard translation model, trained on patent reactions, can predict products at 90.4% top-1 accuracy on the USPTO_MIT benchmark, outperforming the best of twelve human chemists on a set of difficult reactions Molecular Transformer for Chemical Reaction Prediction and Uncertainty Estimation. But virtually every model in this literature uses essentially one hand-picked architecture, and nobody knows whether that choice is special to patent-style chemistry or whether it would survive being moved to a different reaction regime.

Meanwhile, a separate literature on architectural transferability has shown, repeatedly and only in vision, that network wiring is a portable regularizer: connectivity motifs found on small image tasks transfer to new tasks SMASH: One-Shot Model Architecture Search through HyperNetworks, architecture rankings are stable across CIFAR and ImageNet-scale datasets in the 15,625-architecture NAS-Bench-201 lookup table NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search, and population-level design-space analysis shows that spaces dense with good models on cheap proxies stay good at scale On Network Design Spaces for Visual Recognition. None of this has ever been tested on sequence-to-sequence chemical translation — the open-problem list in the transferability literature explicitly notes that cell-based vision spaces "do not generalize easily to autoregressive sequence-to-sequence encoder-decoder designs." This proposal closes that gap by building the missing measurement instrument and then using it.

Core Claim

The claim is twofold. First, we can build a tabular architecture benchmark for chemical sequence-to-sequence models — a chemistry analog of NAS-Bench-201 — at reasonable compute by training thousands of small encoder-decoder variants on a proxy subset of USPTO-50K retrosynthesis and USPTO_MIT forward prediction, with parameter counts and FLOPs strictly controlled. Second, with that table in hand we can answer two questions nobody has: whether architectural rankings transfer across chemical reaction regimes (template-heavy patent retrosynthesis versus route-diverse and temporally split retrosynthesis, and forward versus backward translation), and whether a zero-cost proxy — a formula that scores an untrained network from a single cheap forward pass — can rank chemical translation architectures, something existing proxies were never designed to do. The deliverable is not a new state-of-the-art reaction model; it is the first evidence on whether architecture is a transferable quantity in chemistry, plus a training-free scoring method that would make future architecture choices in this domain nearly free.

The motivation is concrete, not speculative. The domain shows large, unexplained spreads across base architectures: the post-hoc enhancer RetroMPA boosts eight retrosynthesis models by an average of 5.50% top-1 on USPTO-50K, but the gain is almost perfectly anticorrelated with base model quality ($r = -0.96$), with a vanilla Transformer gaining +8.19% while heavily optimized models gain little RetroMPA: A Molecular Property-Aware Auxiliary Framework for Enhancing Retrosynthesis Prediction — base architecture capacity clearly matters, yet is chosen by convention. And the domain's regimes genuinely differ: a survey of the field reports forward prediction near maturity at roughly 91% top-1 for leading models while template-free retrosynthesis stalls between 50% and 65%, an asymmetry attributed to the one-to-many nature of retrosynthesis A Unified View of Deep Learning for Reaction and Retrosynthesis Prediction: Current Status and Future Challenges. Whether the same wiring suits both directions is exactly the kind of question a tabular benchmark answers.

Research Program

The program has three workstreams, each feeding the next.

The first workstream builds the benchmark, which we call ChemSeq2Seq-Bench. The design space is hierarchical rather than cell-based, since the transferability literature itself identifies cell DAGs as the wrong macro structure for encoder-decoder models. Each architecture fixes a hidden width and samples encoder depth (2–6 layers), decoder depth (2–6), attention heads per layer (1, 2, 4, or 8), feed-forward expansion ratio (1, 2, 4, or 8), optional grouped feed-forward layers following SqueezeBERT, which showed that grouped convolutions imported from mobile vision cut Transformer FLOPs roughly threefold without hurting language performance SqueezeBERT: What can computer vision teach NLP about efficient neural networks?, the placement of cross-attention layers (every decoder layer versus every other), and an optional copy mechanism in the decoder, motivated by C-SMILES, where a pointer-generator that copies unchanged molecular scaffolds reached 67.2% top-1 on USPTO-50K with 99.9% validity Copy-Augmented Representation for Structure Invariant Template-Free Retrosynthesis. We sample roughly 5,000 architectures, train each on a fixed 10,000-reaction proxy subset for a few epochs under each of three learning-rate schedules — NAS-Bench-201's own documented failure mode is that a single uniform schedule unfairly penalizes some architectures NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search — and record top-1 exact-match accuracy, SMILES validity, round-trip accuracy, FLOPs, and parameter count. A 500-architecture cohort is additionally trained to convergence and on the full datasets, so that proxy-to-full ranking transfer is itself measured rather than assumed.

The second workstream runs the transferability analysis on the table. The central statistic is the Spearman rank correlation $\rho$ between per-architecture accuracies across two regimes; the field's only precedent says $\rho$ should be high (NAS-Bench-201 found rankings robustly correlated across CIFAR-10, CIFAR-100, and ImageNet-16-120), but that evidence is convolutional-classification-only. We compute $\rho$ across four axes: proxy subset versus full training, USPTO-50K versus the route-diverse (USPTO-rd, rd-1k) and temporally split (USPTO-rt, rt-1k) evaluation sets introduced by Retrosynthesis Prediction Revisited, retrosynthesis versus forward prediction, and, if the curated total-synthesis data can be licensed (its authors note the raw data cannot be publicly released), patent chemistry versus the TotSyn regime, which contains 105,002 reactions and 30,703 templates against USPTO-50K's 2,367 — a thirteenfold difference in reaction-type entropy TotSyn: A Total Synthesis Reaction Dataset for Machine Learning in Organic Chemistry. We also apply the population-level method of Radosavovic et al. — empirical distribution functions with a Kolmogorov–Smirnov distance between design sub-spaces at matched complexity — to ask whether, for example, the sub-space containing the copy mechanism dominates the one without it, in the same way the DARTS space dominated NASNet with $D = 0.51$ On Network Design Spaces for Visual Recognition.

The third workstream evolves a zero-cost proxy for this space, extending the genetic-programning approach of EZNAS, which evolves scoring formulas as expression trees over network tensors and reached a Kendall Tau of 0.65 on NAS-Bench-201 versus 0.57 for hand-designed proxies like NASWOT and Synflow EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring. The key change is the primitive set: EZNAS's primitives assume ReLU-Conv2D-BatchNorm blocks, which do not exist here, so our expression trees operate on encoder self-attention, decoder self-attention, and — critically — cross-attention activation statistics, the tensor that Molecular Transformer for Chemical Reaction Prediction and Uncertainty Estimation showed implicitly learns atom-mapping between reactants and products. We deliberately evaluate the proxy where existing ones are known to fail: the top decile of the ranking, the regime the transferability literature flags as the unresolved weakness of zero-cost scoring.

Baselines

The naive reference point is the conventional choice: the standard Molecular Transformer configuration, trained by us in the same small-compute regime as every other candidate. This "do nothing" baseline answers the question a chemist actually asks — what fraction of the sampled space beats the default wiring at matched parameters, and by how much. If the distribution shows the default sitting near the median, architecture is a live lever in this domain; if it sits at the top decile, that is itself the finding, and it would argue the field's convention is already near-optimal. Both outcomes are informative, and the benchmark reports the whole distribution rather than a single point estimate, following the design-space critique that isolated winners are unreliable On Network Design Spaces for Visual Recognition.

The strongest competitor on the proxy question is EZNAS-A, the evolved vision proxy that holds the best reported ranking consistency on NAS-Bench-201 (Kendall Tau 0.65 on CIFAR-10, with NASWOT and Synflow at 0.57) EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring, together with PLoP's Normalized Feature Norm, a single-forward-pass module-alignment score that beat heuristic LoRA placement on GSM8K (75.03% versus 71.49%) but was developed only for decoder-only models on natural language PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models. We port both to our space unchanged, score the identical candidate pool, and compare Kendall Tau under identical conditions. Our proxy beats them only if it materially exceeds their transferred Tau and, more stringently, if it ranks the top 10% of architectures better than chance-level ordering among that decile. The ablation that isolates our key ingredient runs the same evolutionary search with cross-attention and decoder-side primitives removed, restricting the expression trees to encoder-only statistics — if the full primitive set does not beat this restricted one, the seq2seq-specific contribution is not real and we will say so.

On raw accuracy, the strongest existing numbers are context, not competitors: our small proxy-trained models are not expected to approach the 90.4% top-1 USPTO_MIT forward accuracy of the fully trained Molecular Transformer Molecular Transformer for Chemical Reaction Prediction and Uncertainty Estimation or the 67.2% USPTO-50K retrosynthesis accuracy of C-SMILES Copy-Augmented Representation for Structure Invariant Template-Free Retrosynthesis; they anchor the scale of the gap between the proxy regime and the frontier.

Why not just…?

Why not just use the standard Molecular Transformer and tune it? Because tuning one architecture gives no evidence about transferability, which is the actual question. The transferability literature's central methodological lesson is that single point estimates mislead: Radosavovic et al. showed that spaces which look superior at the point-estimate level can be statistically indistinguishable once complexity confounders are controlled On Network Design Spaces for Visual Recognition, and RetroMPA's near-perfect anticorrelation between base model quality and post-hoc gain shows base architectures in this domain differ in capacity in ways nobody has systematized RetroMPA: A Molecular Property-Aware Auxiliary Framework for Enhancing Retrosynthesis Prediction. That said, the skeptic has a real point that architecture may be a small lever here: if our benchmark's accuracy distribution turns out to be tight at matched parameters, the honest conclusion is that wiring matters less in chemical translation than in vision, and the benchmark's value then rests on the proxy and transfer measurements rather than accuracy headroom. The experiments are designed so that this outcome is a reportable result, not a failure.

Why not just apply PLoP or BIPEFT directly? These are the closest existing systems and we treat them as baselines rather than dismissing them. PLoP scores modules of decoder-only LLMs on natural language and aggregates by module type across layers PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models; BIPEFT searches adapter configurations for T5-class models on classification benchmarks BIPEFT: Budget-Guided Iterative Search for Parameter Efficient Fine-Tuning of Large Pretrained Language Models. Neither touches encoder-decoder generation, SMILES grammar, or cross-attention, and our benchmark is precisely what lets us test whether their scoring signals survive the modality change. If the transferred Normalized Feature Norm ranks our space as well as our evolved proxy does, then the honest finding is that the vision- and NLP-derived signal was already general, and the evolved seq2seq primitives were unnecessary — the head-to-head in the benchmark settles this either way.

Why not just pretrain a bigger chemical language model? Large-scale self-supervised pretraining over molecular corpora — masked modeling over 9.6 billion molecules Language models for the prediction of SARS-CoV-2 inhibitors or autoregressive pretraining for property prediction Inductive transfer learning for molecular activity prediction: Next-Gen QSAR Models with MolPMoFiT — is a genuine threat to the practical value of architecture search, because data scale may dwarf wiring effects. But pretraining confounds architecture with data in a way that makes the transfer question unanswerable: you cannot measure whether wiring helps if every model also carries a different pretraining history. Our benchmark trains from scratch under a fixed data budget to isolate the architectural variable, and we include one fixed-pretraining ablation cohort to check whether architectural rankings shift once a shared pretrained initialization is added. If pretraining flattens the differences, that is a decisive, publishable negative result for architecture search in this domain.

Proposed Experiments and Decisive Checks

Experiment one establishes the benchmark and the proxy-transfer measurement: 5,000 architectures trained on the 10,000-reaction proxy subset, with the 500-architecture convergence cohort reporting Spearman $\rho$ between proxy-trained and fully-trained rankings. The decisive threshold is $\rho > 0.7$ for the proxy setting to be a legitimate stand-in, the standard implicitly set by cross-dataset stability in NAS-Bench-201 NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search.

Experiment two measures regime transfer: rank correlations between USPTO-50K, the USPTO-rd/rt splits of Retrosynthesis Prediction Revisited, USPTO_MIT forward prediction, and TotSyn where licensable, plus Kolmogorov–Smirnov comparisons between design sub-spaces at matched FLOPs. Our working hypothesis — falsifiable and stated in advance — is that macro allocation (encoder-versus-decoder depth, copy-mechanism presence) transfers across regimes while fine-grained choices (head counts, cross-attention placement) do not, with the forward–retrosynthesis axis showing the weakest transfer because the tasks differ in output multiplicity A Unified View of Deep Learning for Reaction and Retrosynthesis Prediction: Current Status and Future Challenges. We will additionally record prediction diversity per architecture, since Retrosynthesis Prediction Revisited found that high single-step accuracy models like Chemformer-Large can lack the output diversity needed for multi-step planning; if architectural choices move diversity more than accuracy, that is a practically important and previously unmeasured axis.

Experiment three evaluates the evolved zero-cost proxy against the transferred EZNAS-A, NASWOT, Synflow, and Normalized Feature Norm baselines on identical candidate pools, reporting Kendall Tau overall and restricted to the top decile. The decisive check is whether the seq2seq-native primitive set beats the encoder-only ablation by a margin that survives seed variation across the evolutionary runs; if it does not, we report the negative result and the benchmark itself remains the contribution. Together, the three experiments produce the first quantitative answer to whether architectural inductive biases — portable across vision datasets for a decade — also travel across chemistry, and a free method for choosing reaction-model wiring without training anything.