Learning from Failed Molecules for Safe, Diverse Molecular Optimization

Generative models for drug discovery discard their most informative outputs. When a molecule generator proposes a candidate that binds its target strongly but fails one toxicity filter, standard pipelines throw it away and resample. Under realistic conditions this waste is expensive: property oracles for binding, toxicity, and quantum chemistry cost real computation or wet-lab time, and practical benchmarks cap evaluation at roughly 10,000 oracle calls (Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization). Meanwhile, two failure modes are well documented on the generation side: reinforcement-learning fine-tuning collapses generators onto a single chemical scaffold family (Pushing Biomolecular Utility-Diversity Frontiers with Supergroup Relative Policy Optimization), and penalty-based handling of safety constraints either suffocates exploration or lets violations through (Safe Reinforcement Learning via Constrained Markov Decision Processes).

The proposal in one breath: fine-tune a fragment-level discrete diffusion model — a generator that builds molecules by iteratively filling in masked fragments rather than appending atoms one at a time — under hard safety constraints enforced by adaptive dual variables instead of fixed penalty weights, reward set-level diversity only among feasible candidates, and retain every failure: near-misses train a feasibility-boundary model, get repaired by minimal fragment edits, and a small probing policy deliberately keeps generating borderline failures so the boundary model does not decay. Each of the four research directions plays a distinct role: safe exploration supplies the constrained-optimization formulation, learning from failure supplies the mechanism, the diversity-utility tradeoff supplies the set-level objective, and chemical space is the application regime where all three must operate under a strict oracle budget.

Where Prior Work Stops

Three papers each connect two directions and leave the rest open. SGRPO (Pushing Biomolecular Utility-Diversity Frontiers with Supergroup Relative Policy Optimization) fixes mode collapse by computing, for each generated molecule, its leave-one-out marginal contribution to the diversity of its batch, then combining that credit with utility through a fixed scalarization weight. It enforces no hard constraint: an ADMET toxicity threshold cannot be guaranteed, and failed candidates are simply absent from its credit computation. CFO (Constrained Flow Optimization via Sequential Fine-Tuning for Molecular Design) enforces hard constraints with an augmented Lagrangian — dual variables that grow when constraints are violated — baked into a flow-matching generator's weights, but it optimizes single molecules with no diversity objective, uses violations only to update dual variables rather than to learn a boundary representation, and its own authors report that structural validity degrades during aggressive fine-tuning. FDPI (Breaking Safety Paradox with Feasible Dual Policy Iteration) identifies the safety paradox — as a policy becomes safe it stops producing violating samples, which starves and degrades the feasibility critic until late-stage violations spike — and fixes it with an adversarial dual policy that deliberately probes the boundary, but only in continuous robotic control. Two further results constrain the design. Static penalty weights are ruled out empirically: unconstrained penalty baselines in CFO's synthetic benchmarks suffered violation rates of 21–49% versus 8.5% for constrained optimization, and manual penalty tuning was roughly 18x more expensive computationally. And the boundary-seeking principle of BSPG (Boundary-Seeking Policy Gradient for Safe Reinforcement Learning) — the provable fact that the optimal constrained policy sits exactly on the constraint boundary — collides with chemical reality: molecular property landscapes are punctuated by activity cliffs, where a single atom substitution collapses bioactivity or safety (Pandey et al.). Hugging a boundary that is discontinuous is a hazard, and no existing method reconciles the two.

Core Claim

The claim is that a molecular generator can be simultaneously constraint-satisfying, structurally diverse, and sample-efficient if and only if failures are treated as first-class training data. Concretely: under a 10,000-oracle budget with hard safety constraints, a system that (i) enforces constraints with adaptive dual variables, (ii) computes diversity credits only over the feasible subset of each batch, and (iii) trains a boundary model and a repair policy on retained near-miss failures, will reach a higher feasible-score area-under-the-curve than the same backbone with any two of those three components, while maintaining violation rates at or below constrained fine-tuning baselines.

Research Program

The backbone is GenMol (GenMol: A Drug Discovery Generalist with Discrete Diffusion), a pretrained discrete diffusion model over SAFE representations, in which molecules are strings of BRICS-derived functional fragments that can be masked and resampled independently. This choice is deliberate: fragment-level remasking gives a natural, validity-preserving edit operator, addressing the validity degradation CFO observed under atom-level flow fine-tuning, and GenMol is the strongest utility generator on the PMO benchmark (Sum AUC of 18.362 versus roughly 16.9 for the best specialized baselines, per its own report).

Four components are built on this backbone. First, the constrained objective replaces SGRPO's scalarization with per-constraint dual variables updated from batch violation rates, CFO-style. The training advantage for candidate $x_i$ in a batch is

$$A(x_i) = u(x_i)\,\mathbb{1}[c(x_i)\le 0] \;+\; \alpha\,\Delta\mathrm{Div}_F(x_i) \;-\; \sum_j \lambda_j\, c_j(x_i),$$

where $u$ is the oracle utility, $c_j$ is the (signed) violation of constraint $j$, $\lambda_j$ are dual variables grown by an augmented-Lagrangian rule, $\mathbb{1}[\cdot]$ is the indicator that $x_i$ is feasible, and $\Delta\mathrm{Div}_F(x_i)$ is SGRPO's leave-one-out diversity credit — computed over the feasible subset $F$ of the batch only, so diversity is rewarded along the feasible boundary rather than paid out to toxic explorers. Second, a failure bank stores every evaluated molecule with its per-constraint margins. A boundary model, trained contrastively on feasible versus infeasible molecules, predicts per-constraint violation. Near-misses — high utility, exactly one constraint slightly violated — are repaired by masking the fragment most responsible (attributed via per-fragment ablation against the boundary model) and resampling, and the (failed, repaired) pair trains the policy with hindsight-style relabeling, adapting to chemistry the idea that a failed trajectory relabeled under its achieved outcome remains useful training signal (AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling). This is the learned, continuous analog of the bridge-predicate mechanism in Learning What's Missing: Failure-Driven Skill Discovery via Predicate Bridges, which uses an LLM to name the symbolic gap behind a planning failure; here the "gap" is a fragment edit and it is extracted from data, not from a predefined predicate vocabulary. Third, a dual prober — a lightly perturbed sampling configuration of the same model, KL-bounded to the main generator — deliberately samples near the boundary model's uncertain regions; its outputs feed only the failure bank and boundary model, never the utility objective, transplanting FDPI's antidote to the safety paradox into discrete chemical space. Fourth, a cliff-aware margin: the boundary model estimates local cliff risk as the frequency of feasibility-label flips among a molecule's nearest neighbors in the bank, and the effective constraint threshold is tightened proportionally, so the policy hugs the boundary only where the boundary is smooth — an explicit, testable departure from BSPG's unconditional boundary optimality.

Proposed Experiments

The primary evaluation is a constrained variant of the PMO protocol (Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization): its 23 tasks (DRD2, GSK3β, JNK3, QED, and GuacaMol multi-property objectives) augmented with hard constraints drawn from the Therapeutics Data Commons oracles (Therapeutics Data Commons: Machine Learning Datasets and Tasks for Therapeutics) — an hERG channel-blockage threshold, a solubility window, a synthetic-accessibility cap, and banned reactive substructures. The 10,000-oracle budget covers constraint evaluations as well as utility, so post-hoc filtering pays a real price. Primary metric: PMO's top-10 average AUC restricted to feasible molecules. Secondary: violation rate, mean pairwise Tanimoto distance among the top-10 feasible set, and the utility-diversity hypervolume over the run. TDC's reported generalization drop on temporal drug-target splits (Pearson correlation falling from roughly 0.7 in-distribution to 0.4 out-of-distribution) motivates a held-out oracle check that the boundary model has not just memorized its training distribution.

A second experiment replicates CFO's GEOM Drugs setting — maximizing dipole moment while holding molecular energy below the −80 Ha stability threshold, where CFO improved reward from 6.55 to 8.39 Debye — under matched compute, additionally reporting feasible-set diversity and validity rate, where CFO's noted validity degradation gives us a concrete target to beat. A third experiment is the causal test of the safety paradox in chemical space: track the violation-sample ratio in the failure bank and the boundary model's error over training, with and without the dual prober. The FDPI analogue predicts that without the prober the ratio decays below roughly 1% late in training and boundary error spikes; FDPI maintained a ratio about 10x higher than its non-adversarial baseline and achieved near-zero violations across all 14 Safety-Gymnasium environments, and we test whether the same dynamics and remedy carry over. A fourth ablation isolates the failure-reuse ingredient by comparing three levels: discard failures (dual updates only), failures train the boundary model, and failures additionally train the repair policy.

Baselines

The naive reference point is unconstrained GenMol with post-hoc rejection filtering: generation at the base model's native speed, every constraint check charged to the budget, no learning from what is rejected. It measures the pure cost of wasted inference and is expected to exhaust its budget fastest. The strongest simple classical competitor is Graph GA with constraint scores folded into its fitness function, since the original PMO study found REINVENT ranked first overall with Graph GA close behind, and the GuacaMol study found Graph GA models generally performed best across goal-directed benchmarks (GuacaMol: Benchmarking Models for de Novo Molecular Design); no Sum AUC figure for REINVENT appears in the file, so the ranking, not a number, is the reference. The strongest constrained method is CFO's augmented-Lagrangian fine-tuning; because CFO targets continuous flow models, we reimplement its exact dual-update rule on our diffusion backbone for the head-to-head, and calibrate our reimplementation against CFO's published GEOM Drugs numbers. The strongest diversity method is SGRPO fine-tuning of GenMol, with constraints scalarized into its utility term and a post-hoc filter appended — this assembled competitor is the closest existing recipe for the combined problem and therefore the bar to clear. All methods receive the identical 10,000-oracle evaluation budget, identical oracles and thresholds, the same backbone where one exists, and an equal hyperparameter search budget (a fixed number of tuning runs per method, charged to a separate budget). Beating the assembled SGRPO-plus-filter baseline means higher feasible top-10 AUC with a violation rate at or below every constrained baseline and top-10 feasible-set diversity at least matching SGRPO's. Two ablations isolate the proposal's key ingredients: the full system minus failure reuse (constraints and feasible-restricted diversity only) separates the learning-from-failure contribution, and minus the dual prober separates the safety-paradox contribution.

Why not just…?

Why not just add penalty terms to the reward? This is the first thing a reviewer will suggest, and the literature gives a concrete answer: fixed penalties produce either the "safe but useless" frozen policy or routine violations (Safe Reinforcement Learning via Constrained Markov Decision Processes), and in CFO's synthetic benchmarks penalty-based baselines violated constraints 21–49% of the time versus 8.5% for constrained optimization, with manual tuning costing roughly 18x more compute. Still, we do not argue this rhetorically: the scalarized variant is run as a baseline under identical budget, so if adaptive dual variables confer no measurable advantage over a well-tuned penalty at matched search budget, that will show directly and the claim narrows accordingly.

Why not just use Graph GA or REINVENT? This is a genuine threat. The original PMO study's own headline was that simple classical methods beat sophisticated deep generators under tight budgets, and GenMol's later win (18.362 versus ~16.9) shows the deep side is not automatically ahead. Our response is empirical, not dismissive: those rankings were obtained without hard constraints, where a generator's failure rate never counts against it. Under the constrained protocol, methods that cannot learn from failures must re-buy the same violation information repeatedly, and SGRPO's analysis shows REINVENT-style fine-tuning suffers scaffold collapse. Graph GA with penalties runs as a baseline; if it still wins under constraints, we report that as the finding.

Why not just keep a memory of past molecules and penalize similarity to it? SGRPO's authors analyzed exactly this memory-assisted design and found it causes distributional drift — the generator wanders into unvisited but low-utility regions purely to be different — and that batch-level leave-one-out credit achieves better utility-diversity Pareto metrics. The file reports no numerical hypervolume or distance-to-ideal-point values for that comparison ("no reported result found on Lacuna for SGRPO's hypervolume on CrossDocked2020"), so rather than lean on the qualitative claim we rerun the memory-assisted variant ourselves under the constrained protocol and report the numbers.

Why not just generate freely and filter afterwards? Rejection sampling is the industry default, and it fails here for three compounding reasons: each filtered candidate consumed oracle budget that taught the policy nothing; the generator never learns why molecules fail, so it keeps proposing borderline structures — the chemical version of the safety paradox FDPI diagnosed; and CFO's authors note that inference-time guidance and filtering slow sampling, whereas internalized constraints generate valid candidates at native speed. The unconstrained-plus-filter baseline quantifies all three costs in oracle calls and wall-clock time.

Decisive Checks

Three results would decide the proposal's fate. The paradox experiment is falsifiable in the most useful way: if the violation ratio in the failure bank does not decay and boundary error does not spike without the prober, then the safety paradox does not transfer to discrete chemical space and the prober should be deleted — a clean negative result worth reporting either way. The cliff-margin ablation settles the BSPG tension: if boundary-hugging does not increase violations on cliff-prone tasks, the margin mechanism is unnecessary and the simpler design wins. And the failure-reuse ablation is the heart of the core claim: if the boundary model and repair training add nothing beyond dual variables and feasible-restricted diversity, the proposal reduces to an engineering combination of CFO and SGRPO, and we would say so. Success looks like this: feasible top-10 AUC above the assembled SGRPO-plus-filter competitor across the constrained PMO tasks, violation rates at or below the CFO-rule baseline, validity preserved by fragment remasking, and every one of the 10,000 oracle calls — successes and failures alike — contributing to what the generator knows.