Chemoselectivity as a Verifiable Reward: RL for Late-Stage Functionalization Design

Generative models that design drug candidates with reinforcement learning face a verification problem, not a generation problem. The reward signal decides what the policy learns, and in molecular design that reward must check two things at once: does the molecule fit the protein pocket in three dimensions, and can it actually be made? Today these checks are done by tools that are either slow or wrong. Docking engines and 3D alignment tools sample thousands of conformations and run iterative coordinate optimization for every candidate the policy proposes — the same per-instance bottleneck that made early neural 3D reconstruction impractical in vision. And retrosynthesis checkers such as AiZynthFinder, which are treated as flawless yes/no oracles in current pipelines, only match molecular graphs against generic 2D reaction templates; they are blind to whether an unprotected amine, a sulfur atom, or a radical trap elsewhere on the molecule will poison the reaction.

This blindness matters most in late-stage functionalization (LSF) — the strategy of installing the final, sensitive groups (a fluoroalkyl handle, a cyano group, a deuterium label) onto an already-complex, unprotected molecule in the last steps of a synthesis. LSF success is governed by functional group tolerance and chemoselectivity: real photoredox and electrochemical protocols tolerate natural products like Lanosterol and Estrone, yet fail on styrene derivatives, sulfur-containing auxiliaries, or electron-poor heterocycles for reasons a 2D template cannot see (Integrated redox-active reagents for photoinduced regio- and stereoselective fluorocarboborylation; Chelation-Assisted Regioselective C−O Bond Cleavage Reactions of Acetals by Grignard Reagents; Ligand- and Additive-Free 2-Position-Selective Trifluoromethylation of Heteroarenes Under Ambient Conditions). The proposal here is to make chemoselectivity itself the verifiable reward: train a fast, condition-aware tolerance and site-selectivity model on curated LSF substrate-scope data (including reported failures), use it as the second stage of a staged verifiable reward, correct its known noise explicitly during policy optimization, and amortize all 3D geometry into a feed-forward pass so the loop stays fast.

Core Claim

Current synthesizable-molecule generators — 3DSynthFlow and its successor SYCLE — verify synthetic feasibility with AiZynthFinder and assume any synthon with a valid attachment handle couples with perfect selectivity (Compositional Flows for 3D Molecule and Synthesis Pathway Co-design; Synthon Contrastive Learning for Synthesizable 3D Molecule Generation). I claim that an RL policy whose reward includes a chemoselectivity stage — a learned predictor of whether a proposed LSF step tolerates the molecule's other functional groups and hits the intended site under stated conditions — will produce routes that are not just formally retrosynthesizable but survive complex, unprotected substrates, at rollout speeds comparable to latent-embedding methods. Because this tolerance predictor is itself an imperfect verifier, I further claim its false-positive and false-negative rates must be measured and corrected during training, importing the noisy-verifier machinery of Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers into chemistry for the first time. The per-instance 3D overhead enters as a hard constraint: every rollout must get an explicit pocket pose from a single forward pass (plus a small fixed number of refinement steps), never from a docking search.

Research Program

The work has four components. First, a tolerance corpus. The substrate-scope tables of modern LSF papers are unusually information-dense: they report yields on deliberately complex, unprotected substrates and — crucially — name their failures. I will extract positives, negatives, and condition metadata from the scopes in the file: the fluorocarboborylation scope (yields 60–93%, natural-product tolerance up to 10 mmol, but benzylic-radical quenching on styrenes) (Integrated redox-active reagents for photoinduced regio- and stereoselective fluorocarboborylation), the paired-electrolysis amination scope (>70 combinations, 38–84% yield, with the Frontier Molecular Orbital analysis explaining why α-CF₃ radicals need paired redox control — a 2.72 eV SOMO–LUMO gap versus 1.36 eV for tert-butyl) (Electrochemical deoxygenative amination of stabilized alkyl radicals from activated alcohols), the C2-trifluoromethylation scope (62–90% yields, with unreactive cyano- and nitro-substituted indoles as true negatives) (Ligand- and Additive-Free 2-Position-Selective Trifluoromethylation of Heteroarenes Under Ambient Conditions), and the Grignard acetal-cleavage scope with its sulfur exclusions (Chelation-Assisted Regioselective C−O Bond Cleavage Reactions of Acetals by Grignard Reagents). This directly attacks the negative-data deficit that the selectivity-prediction literature identifies as the reason current models "suffer from severe false positive rates when applied to promiscuous or non-selective substrates" (Computational tools for the prediction of site- and regioselectivity of organic reactions). That review is the motivating prior work: it shows supervised models like Regio-MPNN reach 96% accuracy on cross-coupling site prediction, but only as passive, offline scorers with no reaction conditions, no negative data, and no role in any generative loop.

Second, the verifier. I will train a graph-based model that takes a proposed reaction step plus its conditions (catalyst class, redox mode, solvent class) and outputs a site-selectivity distribution over candidate bonds and a probability that every other functional group survives. This is the chemical analogue of the staged rewards in Solver-Informed RL, where a cheap syntactic stage gates an expensive semantic stage: stage one is a template match (the AiZynthFinder role), stage two is the tolerance check. Stage two must run in milliseconds, because it executes on every policy rollout.

Third, noise correction. The tolerance verifier will have real false-negative rate ρ₁ (it will reject legitimate novel LSF steps) and false-positive rate ρ₀ (it will bless steps that fail on substrates unlike its training distribution). I will treat it as the stochastic channel formalized by Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers, which documents false negatives up to 38% and false positives from 35% to 68% in common automated verifiers, and apply its forward correction with an appeals mechanism. Writing r(τ) ∈ {0,1} for the raw verifier verdict on rollout trajectory τ, the de-biased reward used for the policy gradient is

$$\tilde{r}(\tau) = \frac{r(\tau) - \hat{\rho}_0}{1 - \hat{\rho}_0 - \hat{\rho}_1},$$

where ρ̂₀ and ρ̂₁ are estimated per reaction class (not globally, addressing that paper's own static-noise limitation) by periodically auditing rejected trajectories with a slower, higher-fidelity appeal — for site selectivity, occasional quantum-chemical descriptor calculations; for pose geometry, occasional full docking runs. The forward-correction variant (reweighting sample importance by ρ̂₁, which that paper showed to be more numerically stable, lifting Math500 Pass@1 from 41.1% to 43.1% on Qwen2.5-7B) will be the default, with the inversion above as the ablation.

Fourth, amortized 3D. The pocket-fit component of the reward must not run a conformer search per rollout. Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance showed the speed win of removing test-time 3D alignment (1.5x to 3.3x faster RL with feed-forward embeddings) but its latent distance outputs no coordinates — you cannot inspect a steric clash. I will instead port the pattern from GS-LRM: a transformer that regresses explicit 3D atomic coordinates of the intermediate molecule inside the pocket in one forward pass (GS-LRM does this for scene Gaussians in 0.23 s on one A100), with a ReSplat-style option of a small fixed number of recurrent refinement steps in which pose-error features are fed back to correct coordinates without any gradient descent (ReSplat: Learning Recurrent Gaussian Splats). Unlike the latent-embedding distance, the output is an explicit pose, checkable by PoseCheck for clashes and hydrogen bonds — the interpretability PED explicitly lacks, and the same validation tool 3DSynthFlow already uses (Compositional Flows for 3D Molecule and Synthesis Pathway Co-design). A secondary extension relaxes SYCLE's fixed breadth-first assembly order so the policy learns to schedule sensitive functionalizations last — the actual logic of LSF — but this is deliberately the smallest piece of the program, not its core.

Baselines

The naive reference point is post-hoc filtering: generate pocket-optimized candidates without any synthesis-aware reward, then discard what AiZynthFinder rejects. This is the regime 3DSynthFlow was built against, and the attrition is documented: its AiZynthFinder success rate of 36.1% on CrossDocked2020 nearly doubles to 68.8% when synthesis awareness moves inside the loop (Compositional Flows for 3D Molecule and Synthesis Pathway Co-design; Synthon Contrastive Learning for Synthesizable 3D Molecule Generation). I will run this reference with the same Enamine building-block library and the same number of sampled molecules as every other arm, to quantify how much of the gain comes simply from filtering earlier.

The strongest competitor is SYCLE (Synthon Contrastive Learning for Synthesizable 3D Molecule Generation), which reported an AiZynthFinder success rate of 68.8% on CrossDocked2020, an average of 1.58 synthesis steps per molecule (against 2.17–3.74 for prior baselines), and the best Vina docking scores across all 15 LIT-PCBA targets, including −12.10 kcal/mol on ADRB2 versus a previous best of −11.96. It is the strongest because it already fixes much of the two-stage attrition and the one-step geometric myopia. I will rerun SYCLE (or its released model where available) on identical data splits, compute, and tuning budget, and evaluate all arms on identical metrics. Beating it means: comparable or better AiZynth success rate and Vina score, while achieving materially higher tolerance-aware route validity — the fraction of generated LSF steps whose predicted site and survival agree with held-out experimental scope data, a metric on which no prior system reports any number. If SYCLE matches my pipeline on that metric too, the proposal's value collapses to a speed claim, and I would report it as such.

Two ablations isolate the ingredients. The clean-verifier ablation removes the tolerance stage and the noise correction, rewarding only template-matched retrosynthetic success with the same feed-forward pose geometry — it isolates the chemoselectivity-aware reward. The oracle-geometry ablation swaps the feed-forward pose predictor for a true docking run per rollout — it isolates the amortization, at the cost the PED paper already quantified as at least a 1.5x to 3.3x slowdown (Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance).

Why not just…?

Why not just generate in 2D and filter afterwards? The evidence against decoupling is direct: 3DSynthFlow reports a 4.2x sampling-efficiency gain over 2D synthesis-only baselines precisely because the model never wanders into chemical space that a later filter would discard, and the AiZynth success gap between decoupled-style baselines (36.1%) and joint generation (68.8%) is the same story in a single number (Compositional Flows for 3D Molecule and Synthesis Pathway Co-design; Synthon Contrastive Learning for Synthesizable 3D Molecule Generation). A tolerance check applied only post-hoc would inherit that attrition, and worse: tolerance failures are exactly the candidates that pass cheap filters and die expensively at the bench.

Why not just use 2D fingerprints and rule-based retrosynthesis? This is a real threat to the speed claim, because 2D methods are fast and well established. But the PED paper already showed 2D fingerprints are the weakest similarity option — its MoLFormer embedding distance reached an average enrichment factor of 4.53 on LIT-PCBA, "outperforming traditional 2D fingerprints" (Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance) — and the selectivity review states plainly that 2D rule sets cannot predict failure from competing reactive sites, solvents, or catalyst deactivation (Computational tools for the prediction of site- and regioselectivity of organic reactions). Since chemoselectivity is the entire object of the reward, 2D rules are not a cheaper version of this proposal; they compute a different, mostly orthogonal quantity. I will still run this baseline, because if it matched my tolerance stage on held-out scope data, that would be a decisive negative result worth reporting.

Why not just run Vina or DFT at test time for maximum fidelity? Per-instance optimization is the most accurate option and the honest upper bound on verifier quality, so I include it as the oracle-geometry ablation rather than argue it away. The cost is documented twice over: per-molecule 3D alignment slowed RL rollouts by the 1.5x–3.3x factor the PED paper measured when removing it (Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance), and the selectivity review notes quantum-mechanical site calculations are too expensive for high-throughput use, which is why Regio-MPNN exists at all (Computational tools for the prediction of site- and regioselectivity of organic reactions). The right use of the slow oracles in this proposal is not per-rollout but as the appeals layer: sparse, high-fidelity audits that calibrate the fast verifier's noise rates — exactly the role the auxiliary "TinyV" auditor plays in the imperfect-verifier paper (Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers). The decisive check is whether the corrected fast verifier recovers oracle-level policy performance at a small fraction of the compute.

Why not just do supervised fine-tuning on literature reactions, with no RL? This is the most serious alternative, and I want to be honest about it: the tolerance model itself is trained supervised, and Regio-MPNN's 96% cross-coupling accuracy shows supervised selectivity prediction can be strong (Computational tools for the prediction of site- and regioselectivity of organic reactions). The rebuttal is about where the model sits. A supervised scorer ranks molecules; it does not search route space, and the one-shot generation it would rank is precisely the decoupled pipeline whose attrition the first "why not" already covers. Moreover, the review identifies what supervised selectivity models lack — negative data, reaction conditions, cross-class generalization — and my corpus construction addresses all three before any RL happens, so the supervised component is a contribution independently of the policy. The experiment that settles it: compare route-level tolerance validity (not molecule-level scores) for SFT-ranked one-shot generation versus the RL policy under identical data and compute.

Proposed Experiments

The evaluation runs on three planes. Plane one is the existing benchmarks, so numbers are comparable: CrossDocked2020 for pocket-conditional generation with AiZynthFinder success rate, average synthesis steps, and Vina score, and the full 15-target LIT-PCBA suite for docking and enrichment, using the Enamine catalog as the shared building-block space (Synthon Contrastive Learning for Synthesizable 3D Molecule Generation; Compositional Flows for 3D Molecule and Synthesis Pathway Co-design). The headline target is SYCLE's 68.8% AiZynth success rate on CrossDocked2020 and its −12.10 kcal/mol ADRB2 Vina score on LIT-PCBA.

Plane two is new and, I believe, the decisive one: tolerance-aware route validity on held-out LSF scope data. Using leave-one-substrate-out splits of the extracted corpus — the fluorocarboborylation, paired-electrolysis, and C2-trifluoromethylation scopes — the tolerance verifier must predict both site (which bond reacts, scored against reported regioisomers) and survival (yield above or below threshold, scored against reported outcomes including the named failures). No reported result found on Lacuna for any system on this metric; establishing a number here is itself a contribution. Site-selectivity accuracy should be benchmarked against the Regio-MPNN 96% cross-coupling figure as the supervised ceiling (Computational tools for the prediction of site- and regioselectivity of organic reactions). Generated routes will additionally be checked with PoseCheck for pose-level validity, matching 3DSynthFlow's geometric evaluation.

Plane three is verifier measurement and speed. I will measure AiZynthFinder's own false-negative rate on LSF steps — no reported result found on Lacuna for this — by running it on the scope corpus's successful reactions, and measure the tolerance verifier's ρ₀ and ρ₁ via the appeals audits, reporting corrected-uncorrected policy performance in the style of the Math500 experiment (41.1% to 43.1% Pass@1 under noise) (Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers). Wall-clock per rollout will be reported against the Vina-per-rollout oracle arm, with the PED paper's 1.5x–3.3x speedup as the reference point for what amortization bought in a comparable setting (Advancing Ligand-based Virtual Screening and Molecular Generation with Pretrained Molecular Embedding Distance). The single most decisive check is a triangle: if tolerance-aware routes beat the clean-verifier ablation on held-out scope validity, and the corrected verifier beats the uncorrected one at high measured noise, and the feed-forward pose arm stays within the oracle arm's docking-quality band at a fraction of its cost, then chemoselectivity-as-verifiable-reward is doing work that no existing component does alone. If any leg fails, the ablations say which assumption to drop.