Reward Hacking as a Generalization Gap: OOD-Aware Surrogates for Generative Lead Optimization

When a machine learning model is trained and tested on the same kind of data, it can look nearly perfect and still be useless in the wild. Audio deepfake detection is the cleanest documented example: detectors that reach error rates near 1% on studio-quality benchmark speech degrade by 200% to 1000% in relative error when tested on unconstrained real-world audio, with some models doing no better than chance Does Audio Deepfake Detection Generalize?. The diagnosed cause is that the models memorize shallow fingerprints of the specific synthesis engines they saw during training, instead of the signal that actually distinguishes real from fake. This proposal argues that generative lead optimization — the computational loop that iteratively rewrites candidate drug molecules to improve potency and drug-like properties — suffers from the same failure, with worse consequences, and that the fixes proven in audio forensics transfer to chemistry in a way no one has yet tested.

The connection is concrete rather than metaphorical. Modern molecular generators, whether reinforcement-learning scaffold hoppers like RuSH or fragment-based samplers like the GFlowNet agent of GFlowNets for Lead Optimization, do not measure real biological activity; they query a surrogate model — a regression or classification model trained on past assay data — and optimize its predictions. But the DrugOOD benchmark showed that such predictors collapse precisely where generators push them: under scaffold shift, where test molecules have core structures unlike anything in training, standard training achieved only 68.74% out-of-distribution AUC versus 92.75% in-distribution AUC on the size split DrugOOD. A generator that hops scaffolds is therefore actively steering into the region where its own reward signal is unreliable, and it will happily exploit the surrogate's errors — reward hacking — while reporting excellent predicted potency.

Core Claim

The generalization gap in deepfake detection and the reward-hacking failure in lead optimization are the same phenomenon — shortcut memorization of domain identity instead of causal signal — and the countermeasures validated in audio forensics, adapted to molecules, make generative lead optimization robust where it currently fails silently. Specifically, I propose to build a surrogate affinity scorer whose representation is (a) organized as prototypes in hyperbolic space along the natural chemical taxonomy, following Poin-HierNet, and (b) whitened to suppress scaffold-identity dimensions, and to couple it to the generator through a reliability-tempered reward plus a cheap source-free online adaptation step adapted from TAB. The deliverable is not a better static predictor but the first closed-loop lead optimization pipeline that knows when its own scoring is untrustworthy and degrades gracefully instead of hacking.

Why These Two Fields Share a Failure Mode

The audio literature gives a precise anatomy of the gap, and each element has a molecular counterpart. Detectors latch onto low-level vocoder fingerprints, exactly as molecular predictors latch onto scaffold-specific structural features rather than the pharmacophore interactions that actually cause binding — the shortcut that TAB explicitly targets with random atom masking, and that MoSIR targets by splitting embeddings into stable semantic prototypes and noisy environment residuals. Channel and codec variation in audio corresponds to assay-condition and scaffold-family variation in chemistry, which is exactly how DrugOOD constructs its hardest splits DrugOOD. Most tellingly, Poin-HierNet showed that the forgery world is a tree — attack class, then synthesis engine, then codec — that Euclidean geometry embeds with severe distortion, and that moving to the Poincaré ball (a curved space where volume grows exponentially, suited to trees) plus whitening domain-sensitive feature dimensions cut error on ASVspoof 2021 DF from 2.19% to 1.40% EER. Molecular structure is the same kind of tree: Murcko scaffold — the core ring framework of a molecule — branching into ring systems, then into functional decorations. No one has tested whether this geometric diagnosis and fix transfer.

Research Program

The work builds three components, each an adaptation of a proven audio-forensics mechanism to molecular graphs.

The first is a hierarchical hyperbolic surrogate. A graph neural network encoder maps each molecule into the Poincaré ball, where prototypes are organized along the scaffold taxonomy (scaffold family, then ring system, then decoration level), trained with prototype and hierarchy losses in the style of Poin-HierNet but on ChEMBL-derived affinity data partitioned with DrugOOD's scaffold split. Crucially, following Poincaré Feature Whitening, high-variance embedding dimensions that encode scaffold identity — the molecular analog of codec fingerprints — are masked before prediction. This differs from MoSIR, which enforces prototype invariance in Euclidean space with an expensive adversarial bi-level loop and never feeds a generator; and from ScaffoldGVAE, which uses a flat Euclidean Gaussian-mixture latent for generation with no out-of-distribution safeguard on scoring. My proposal puts the geometry in the scorer, not the generator, because the scorer is where hacking originates.

The second component is the reliability-tempered reward that closes the loop. For a candidate molecule $m$, the generator receives

$$\tilde{r}(m) = w(m)\,\hat{r}(m) + \big(1 - w(m)\big)\,\bar{r}_{p(m)},$$

where $\hat{r}(m)$ is the surrogate's predicted activity, $p(m)$ is the nearest hyperbolic prototype with mean training activity $\bar{r}_{p(m)}$, and $w(m) \in [0,1]$ is a reliability weight that decays with the whitened distance from $m$'s embedding to the prototype set. In-distribution, the surrogate is trusted; deep in novel chemical space, the reward shrinks toward an honest prototype-level prior, so the generator has no error to exploit but still receives a meaningful signal to explore.

The third component is online adaptation. As the generator drifts across scaffold families, the surrogate's normalization layers and prototype means are updated on the generated batch itself using masked-graph, uncertainty-weighted consistency — TAB's technique, but on 2D molecular graphs, removing TAB's requirement of 3D crystal structures or docking for every candidate, which is prohibitive inside a generative loop that proposes thousands of molecules per iteration.

Baselines

The naive reference point is doing nothing: an ordinary empirically-trained graph network scorer plugged into the generator unmodified, which is the de facto design of current pipelines — the GFlowNet agent of GFlowNets for Lead Optimization trusts a random-forest proxy (reported $R^2 = 0.72$, $r = 0.83$ on Caco-2 permeability) at face value. On the surrogate-robustness question, the strongest existing competitor in the file is TAB, the source-free test-time adaptation method of TAB, which on the DrugOOD assay task boosted Pearson's $R$ from 0.119 for a standard model to 0.388; I will run a 2D-graph adaptation of TAB inside the loop so the comparison isolates my representation-level ingredients (hyperbolic prototypes plus whitening) from its adaptation mechanism, which I reuse. The second competitor is MoSIR, the prototype-constraint representation of MoSIR, which reported the highest ROC-AUC across DrugOOD shifts but with no reported result found on Lacuna for its exact DrugOOD AUC values, so I will re-run it under my tuning budget. The off-the-shelf domain-generalization tier — DeepCoral at 67.83% and IRM at 67.74% scaffold OOD AUC, both below ERM's 68.74% DrugOOD — is included for completeness. Finally, I will construct an uncertainty-gated baseline: an ensemble of scorers whose disagreement penalizes the reward, with no representational change; this is the strongest simple alternative and is described in the next section. All methods share the same ChEMBL/DrugOOD data, the same encoder architecture and parameter count, the same random-search tuning budget, and the same generator and compute. Beating the baselines means, on the static side, exceeding 68.74% scaffold OOD AUC by at least three points and matching TAB's adapted correlation gains; on the closed-loop side, cutting the promised-versus-realized reward gap by at least half while retaining at least 90% of the hop rate and diversity of the unguarded pipeline. The key ablation swaps the Poincaré prototypes for Euclidean ones with everything else fixed, and separately toggles whitening and reward tempering, so each ingredient's contribution is measured.

Why not just…?

Why not just train on more data? The file's evidence says scaling does not close this gap. In audio, twelve architectures of increasing capacity all suffered the 200–1000% relative error increase on real-world data Does Audio Deepfake Detection Generalize?; in chemistry, DrugOOD showed that expanding from cleaner to noisier ChEMBL tiers pollutes labels and hurts, and that in-distribution gains (92.75% AUC) coexist with out-of-distribution collapse (66.37%) DrugOOD. The failure is about what is learned, not how much.

Why not just apply IRM, DeepCoral, or DANN? These were already benchmarked on the exact task and lost to plain training: 67.83% and 67.74% scaffold OOD AUC versus 68.74% for ERM, leading the DrugOOD authors to conclude that general-purpose domain-generalization algorithms without biochemical inductive bias are ineffective DrugOOD. Running them again inside a generative loop would not change the scoring failure they already exhibit.

Why not just penalize ensemble disagreement? This is the most serious alternative, and I treat it as a first-class baseline rather than arguing it away. It would likely reduce hacking somewhat, but two things suggest it is insufficient. First, ensemble disagreement is itself miscalibrated under scaffold shift — the models in DrugOOD collapse together, so they can agree confidently while being wrong. Second, gating purely on uncertainty either abstains (starving the generator) or passes candidates through unchanged, whereas TAB showed that adaptation can recover signal in shifted domains (Pearson's $R$ from 0.119 to 0.388) rather than merely flagging them TAB. The decisive experiment E4 below compares disagreement-gating, tempering, and adaptation head to head; if gating matches my full method, the simpler design wins and I will report that.

Why not just use a frozen pretrained foundation model with a probe? The DrugOOD two-tower architecture already used pretrained BERT protein embeddings and still collapsed on OOD splits DrugOOD, and in audio a single frozen wav2vec 2.0 baseline scored 6.59% EER cross-dataset, only dropping to 3.08% when orthogonal structural views were fused with attention Cross-Dataset Fake Audio. Pretraining buys breadth, not invariance to the specific shift a scaffold hop induces.

Proposed Experiments

Experiment E1 tests the surrogate in isolation on DrugOOD's scaffold and assay splits (core annotation tier), reporting AUC against ERM (68.74% scaffold OOD AUC as reported by DrugOOD), DeepCoral, IRM, a TAB adaptation, and MoSIR. Experiment E2 replicates the closed-loop permeability setup of GFlowNets for Lead Optimization — the Caco-2 dataset of 455 matched molecular pairs with logPapp values, a proxy trained on a scaffold-confounded split, and the same fragment-based agent, which reported 95% validity, 100% uniqueness, and an average logPapp shift of -1.5 from the starting molecule — and measures the realized improvement using a holdout honest oracle trained across scaffolds. The headline metric is the gap between promised and realized improvement, plus the hacking rate: the fraction of top-rewarded molecules whose oracle score falls below the starting compound. Experiment E3 scales to scaffold hopping on the five kinase targets (CDK2, EGFR, JAK1, LRRK2, PIM1) used by ScaffoldGVAE, which pre-trained on over 800,000 ChEMBL v31 molecule-scaffold pairs and evaluated hops by MM/GBSA binding energies in the -40 to -60 kcal/mol range with a 92.4% success rate on LRRK2; I compare my OOD-aware scorer against an ERM scorer inside the same hopping pipeline, evaluating post-hop activity retention with the same docking protocol. Experiment E4 is the decisive stress test: I deliberately plant a spurious shortcut — a decorative side-chain feature correlated with the activity label only within training scaffolds — and measure how much of each method's top-scoring output exploits it; this directly adjudicates the disagreement-gating alternative. A small fifth experiment checks mechanism transfer in the reverse direction, applying the masked-consistency adaptation on top of standard self-supervised audio features on the In-The-Wild dataset, where raw architectures degraded to 33.9% EER Does Audio Deepfake Detection Generalize? and Poin-HierNet reported 4.91% Poin-HierNet; it is a sanity check on the shared-failure-mode claim, not a bid for a new audio method.

The proposal succeeds if the whitened hyperbolic surrogate beats the 68.74% scaffold OOD AUC baseline, and if the tempered closed loop converts most of its promised reward into realized activity under honest evaluation while unguarded pipelines inflate and collapse. It fails informatively if disagreement-gating matches it — which would itself establish, for the first time, a quantitative account of reward hacking as a generalization gap in generative drug design.