Learning the Reward Instead of Writing It: Negative-Aware Inverse RL for Budget-Constrained Molecular Optimization

Every AI system that designs molecules faces the same bottleneck: someone must write down, as an equation, what a good molecule looks like. In practice this means hand-combining a docking score, a drug-likeness heuristic, and a handful of penalty terms into one scalar reward — the exact "manual reward engineering" that the reinforcement learning literature identifies as a source of degenerate shortcuts. The failure is well documented in chemistry: agents optimizing a static scoring model drift into regions where the model extrapolates confidently but wrongly, and return bloated, unphysical structures that exploit the equation rather than satisfy the biology (Active Reward Learning; Graph Machine Learning for Design of High-Octane Fuels). Meanwhile, the benchmark that defines practical success — the Practical Molecular Optimization (PMO) suite, which caps each algorithm at 10,000 oracle evaluations across 23 tasks — shows that the current best method is simply REINVENT, a policy-gradient agent with a hand-crafted composite score (Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization). The scoring function is the part nobody has automated.

This proposal argues that the reward itself should be learned from data medicinal chemists already have: libraries of confirmed active ligands and confirmed inactive compounds, organized per target. The tool for this is adversarial imitation learning — training a classifier (a discriminator) to tell expert molecules from generated ones, and using its output as a dense reward — but upgraded in two ways that have never been applied to chemistry. First, following Adversarial Inverse Reinforcement Learning (AIRL), we recover a state-only reward that disentangles "what makes a molecule active" from "what the training set happened to look like," so the learned intent can transfer to new scaffolds and targets. Second, following PN-GAIL, we exploit the negative half of chemical databases — known inactives and decoys — as explicit anti-demonstrations. The resulting system needs no per-target reward equation, and because the discriminator is trained offline on existing libraries, its adversarial training consumes zero oracle calls.

Core Claim

A molecular generator whose reward is a disentangled, negative-aware discriminator reward, gated by the applicability domain of the target's activity model, will match or beat hand-scored reinforcement learning on the PMO benchmark at the 10,000-call budget, while producing far fewer molecules that exploit the scoring model's blind spots, and will transfer to scaffold- and target-shifted settings where hand-crafted scores must be rewritten from scratch.

The closest existing work is MTMol-GPT, which trains a GPT-style generator with GAIL-style discriminators over SMILES/SELFIES tokens for dual-target design (MTMol-GPT: De novo multi-target molecular generation with transformer-based generative adversarial imitation learning). It proves the concept but inherits three documented weaknesses: it uses plain GAIL, which matches the token statistics of the expert set rather than recovering portable intent; it treats every molecule in ExCAPE-DB as a positive exemplar, discarding inactives; and it was never audited under a realistic oracle budget or checked for applicability-domain exploitation. Our proposal differs on all three axes, and the third — sample-budget accounting — is where we suspect the most sobering finding will land.

Research Program

We frame molecule generation as a Markov decision process over SELFIES tokens, with a pre-trained SMILES/SELFIES language model as the initial policy, following the architecture conventions of ACEGEN: Reinforcement learning of generative chemical agents for drug discovery. ACEGEN supplies the policy-gradient machinery (it identifies PPO with experience replay, PPOD, as the most sample-efficient inner loop across the 23 MolOpt tasks) — but its scoring functions are still hand-engineered weighted sums, which is precisely the component we replace.

The learned reward takes the AIRL form. A discriminator scores state transitions with

$$f_{\theta,\phi}(s, a, s') = g_\theta(s) + \gamma h_\phi(s') - h_\phi(s),$$

where $s$ is the partially built molecule, $a$ is the next token, $\gamma$ is a discount factor, $g_\theta$ is a state-only function, and $h_\phi$ is a potential-based shaping term (Learning Robust Rewards with Adversarial Inverse Reinforcement Learning). The design point is that $g_\theta$ alone, evaluated on a completed molecule, is a portable activity reward: in AIRL's robotics validation, the state-only reward survived a morphological change (an Ant with two disabled legs scored 130.3 under AIRL's recovered reward while plain GAIL collapsed to -58.8), precisely because intent had been separated from the data-generating dynamics. The chemical analogue of that test is scaffold and target transfer: train on one target's ligands, deploy the recovered $g_\theta$ where the generator's structural habits no longer match the training set. Plain GAIL-style matching — what MTMol-GPT does — has no mechanism to survive this, and we will measure exactly that.

The second ingredient is the negative set. Chemical databases are unusual in imitation learning: they come with large volumes of confirmed inactives, not just noisy positives. ExCAPE-DB maps compounds to targets with both active and inactive labels (MTMol-GPT draws its positive sets from it). We train the discriminator under a PN-GAIL-style positive-negative objective, which forces the policy to deviate from known bad examples and provably counteracts the preference bias that arises when a high-frequency but undesirable mode dominates the demonstration pool (PN-GAIL: Leveraging Non-optimal Information from Imperfect Demonstrations). In chemistry the negative set is richer than in robotics: inactives, decoys, and pan-assay interference compounds are all labeled anti-demonstrations that a hand-written score can only encode as penalty terms someone had to think of in advance.

The third ingredient is the applicability-domain gate. QSAR models — including the DRD2, GSK3$\beta$, and JNK3 activity models that serve as PMO oracles — are only trustworthy inside the structural envelope of their training data, and they fail silently outside it, returning confident nonsense (Defining the Applicability Domain of Chemical Machine Learning). We gate the learned reward by an online applicability-domain estimate,

$$\tilde{R}(m) = g_\theta(m) \cdot \mathbb{1}\left[d_{\text{AD}}(m) \le \tau\right],$$

where $d_{\text{AD}}(m)$ is a Mahalanobis-type distance of molecule $m$'s descriptors from the demonstration set and $\tau$ is a leverage threshold. This replaces the static KL-divergence "chemical anchor" against a pre-trained prior that ACEGEN uses to prevent syntactic collapse: the KL anchor keeps strings grammatical but does nothing about property-oracle exploitation, which is the failure that actually wastes wet-lab money.

The critical accounting argument, and the answer to the objection that adversarial imitation learning is hopelessly sample-hungry, is that the two sample budgets are separable. The min-max game between generator and discriminator consumes policy-gradient steps and offline demonstration molecules (ChEMBL and ExCAPE-DB contain millions), neither of which costs an oracle call. Oracle calls are spent only when a completed candidate molecule is scored against the task's evaluation model. Under PMO's accounting, adversarial reward co-evolution is free. Whether it is fast enough — whether the generator produces competitive candidates within 10,000 scored molecules — is an open empirical question that, per Gap 4 of the literature, nobody has audited; MTMol-GPT reports distribution matching and docking validation but no budget-constrained comparison, and the PMO study evaluated 25 algorithms with no adversarial-imitation entry.

Baselines

The naive reference point is sampling from the ChEMBL-pre-trained prior with no optimization: it establishes the floor on every task and quantifies how much of each benchmark score is reachable by lucky sampling alone. The strongest competitor is REINVENT with a hand-crafted composite score, which the PMO benchmark ranked number one overall across its metrics, ahead of Graph GA, out of 25 evaluated algorithms at the 10,000-call budget (Sample Efficiency Matters); no raw AUC value for REINVENT is printed in the paper text on Lacuna, so the comparison will be run head-to-head under identical conditions — same 23 PMO tasks, same 10,000-oracle-call cap, same top-10-average AUC metric, and the same tuning budget of one practitioner-week per method, with each method's hyperparameters tuned on three held-out tasks before evaluation. The second baseline is Graph GA, the runner-up on PMO and the top goal-directed method in GuacaMol (GuacaMol: Benchmarking Models for de Novo Molecular Design), run under the same caps. The closest method-family baseline is a reimplementation of MTMol-GPT's plain-GAIL generator with target-specific positive-only discriminators (MTMol-GPT), evaluated for the first time under the PMO budget. Finally, three ablations isolate the proposal's ingredients: the same policy with the learned reward swapped for a static hand-crafted score (isolates learned reward), the learned reward with the applicability-domain gate removed (isolates the gate), and the learned reward with the negative set removed (isolates negative-awareness). Beating REINVENT means matching or exceeding its top-10-average AUC on the majority of the 23 tasks at equal budget, while showing a materially lower rate of applicability-domain violations among top-scoring molecules; beating MTMol-GPT's reimplementation means exceeding it on the same AUC metric and, more importantly, on the transfer tests where plain occupancy matching should break.

Why not just…?

Why not just use REINVENT with a well-tuned hand-crafted score? This is the real threat, and we treat it as the primary baseline rather than arguing around it. The evidence for it is strong: REINVENT ranked first overall on PMO across 23 tasks at the realistic budget, and ACEGEN showed that adding experience replay (PPOD) makes this family even more sample-efficient. But two documented costs are invisible in the AUC metric. First, every new target or multi-target profile requires an engineer to re-derive the composite score, a bottleneck the reward-engineering literature identifies as the reason manual shaping fails to scale across task spaces (Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping). Second, static composite scores are reward-hacked: agents find the mathematical blind spots of the QSAR or docking components and return molecules that score well and work poorly. Our experiments settle both points head-to-head: matched-budget AUC against REINVENT, plus a hacking audit (below) that PMO does not perform, plus the transfer test where REINVENT's score must be rewritten by hand and ours must not. If REINVENT matches us on all three, the proposal should be abandoned.

Why not just use a genetic algorithm? Graph GA is remarkably strong on numerical objectives — runner-up on PMO, best on GuacaMol's goal-directed suite, and its combinatorial variant Typed GA achieved a mean top-1000 Docking-Call Reduction Factor of 21.5x at the 10,000-call budget on SynthonBench (SynthonBench: Benchmarking Sample-Efficient Optimization in Combinatorial Chemical Spaces). But the same benchmarks document its ceiling: in GuacaMol it frequently exploited scoring functions and produced bizarre structures, with the simple SMILES LSTM reaching 77% acceptable compounds where GA quality lagged; on PMO it dominates "isomer" tasks but loses to RL on the similarity tasks that resemble real lead optimization; and below 1,000 evaluations on SynthonBench it was beaten decisively by bandit methods (39.6x efficiency for Thompson Sampling) because recombination needs a population first. We include Graph GA as a baseline because it may well win raw AUC; our claim is about the quality, hacking, and transfer dimensions it does not address.

Why not just do supervised translation on matched molecular pairs? Molecular optimization by capturing chemist's intuition using deep neural networks shows a conditional Transformer trained on ChEMBL matched pairs is excellent at what it sees: 98.50% validity, over 90% of outputs being single sensible transformations, and consistent multi-property success on logD, solubility, and clearance. But it is pure behavior cloning — no exploration, no recovery from drift — and its own authors report performance dropping sharply on unseen starting scaffolds and rare property shifts, the exposure-bias failure mode that adversarial distribution matching exists to fix. It cannot discover chemotypes absent from historical pairs, and it cannot tell us why a transformation works, so nothing transfers. We will run it on the transfer splits as a baseline; we expect it to degrade exactly where our disentangled reward is supposed to hold.

Why not just use bandits over a make-on-demand library? For ultra-tight budgets this is genuinely the right tool: Thompson Sampling topped SynthonBench at 39.6x under 1,000 calls. But the same study shows bandits plateauing past ~2,500 calls, trapped in high-utility neighborhoods, and they can only select from pre-enumerated catalogs — no de novo structures, no learned intent. They solve a different problem (combinatorial selection) and we will say so rather than claim superiority; our evaluation is on PMO's generative tasks, where they are not a natural entry.

Proposed Experiments and Decisive Checks

The main evaluation is the PMO benchmark: all 23 tasks (QED, the DRD2, GSK3$\beta$, and JNK3 QSAR oracles, and the 19 multi-parameter GuacaMol-derived tasks), 10,000 oracle calls each, scored by the area under the curve of the top-10 average score as a function of calls, exactly as defined in Sample Efficiency Matters. Positive demonstrations come from per-target actives in ExCAPE-DB and ChEMBL; negatives come from the corresponding confirmed inactives. The inner loop is PPO with experience replay, the configuration ACEGEN identified as most sample-efficient, so any gain is attributable to the reward rather than the optimizer.

The hacking audit is the check PMO lacks. For each method's final top-10 molecules, we measure the fraction falling outside the oracle's applicability domain (leverage and Mahalanobis distance in the oracle's descriptor space, per Defining the Applicability Domain of Chemical Machine Learning), and we re-score every top molecule with a refreshed oracle retrained with the generated molecules excluded, so that scores inflated by extrapolation artifacts drop. Success is a materially lower AD-violation rate and a smaller refreshed-score drop at equal or better AUC; if the learned reward merely finds subtler blind spots, the audit will catch it.

The transfer test is the AIRL-motivated decisive experiment, in two graded versions. Scaffold transfer: hold out an entire chemotype from the demonstration set at training time and ask each method to optimize activity on molecules of that chemotype. Target transfer: train the discriminator on one target's active/inactive data and deploy the recovered $g_\theta$ as the reward for a second, related target, counting how many oracle calls on the new target are needed to reach a fixed score threshold versus a hand-scored baseline that must be re-derived. The AIRL precedent (130.3 versus -58.8 under morphological damage) predicts plain GAIL-style matching collapses here while the disentangled reward survives; if it does not, the disentanglement claim is falsified and the project reduces to a negative result worth reporting, since MTMol-GPT-style matching is the only form of imitation currently in the chemical literature.

Finally, an honest secondary study addresses the sample-efficiency question directly: a budget sweep at 100, 1,000, and 10,000 oracle calls on a subset of tasks, comparing against the budget-dependent regime changes documented on SynthonBench (bandits dominant below 1,000 calls, Typed GA at 21.5x DCRF at 10,000) to locate where, if anywhere, adversarial reward learning earns its keep. The working hypothesis is that offline discriminator training front-loads the cost, so the learned-reward agent should be most competitive at small budgets — and if instead it needs the full 10,000 calls just to stabilize the min-max game, that too is a finding the field currently has no number for.