Flatness-seeking optimizers like Sharpness-Aware Minimization (SAM) train a network by, at every step, first finding a worst-case parameter shift within a ball of radius $\rho$ and then updating the weights using the gradient at that shifted point. The payoff is generalization: models that sit in broad valleys of the loss landscape tolerate the small mismatches between training and test data. The catch, well documented in the optimization literature, is that $\rho$ is brutally hypersensitive — too small and the method collapses back to ordinary training, too large and the adversarial search probes beyond the local valley into unrelated terrain, destabilizing convergence. Chemistry has the mirror-image problem. Molecular property landscapes are punctuated by activity cliffs: pairs of molecules differing by a single atom or methyl group whose potency differs by an order of magnitude or more. A model that is uniformly flattened is, by construction, a model that risks smoothing away exactly the sharp, real, biologically decisive boundaries that matter most.
This proposal argues that these two sensitivities are two faces of one problem, and that the perturbation radius should not be a single global constant but a function of where in chemical space each training molecule sits. The core deliverable is a diagnostic study plus a method — Cliff-Adaptive Perturbation Scaling (CAPS) — that scales the per-example perturbation radius down near suspected activity cliffs and up in chemically smooth regions, and that is evaluated against the cliff-specific error metrics that existing benchmarks now provide.
Core Claim
The claim has three parts. First, as a matter of measurement: nobody has mapped how the weight-space perturbation radius $\rho$ affects prediction error specifically on cliff compounds versus non-cliff compounds. MoleculeACE, which benchmarks 24 algorithms across 30 ChEMBL targets and reports a cliff-specific error ($\text{RMSE}_{\text{cliff}}$) alongside overall RMSE, exists precisely to expose this kind of blind spot Exposing the Limitations of Molecular Machine Learning with Activity Cliffs), yet none of the sharpness papers in this literature have been run through it. Second, as a hypothesis: the radius that minimizes overall RMSE and the radius that minimizes $\text{RMSE}_{\text{cliff}}$ diverge, because a uniform flatness budget spends most of its regularization pressure on the large-gradient directions that define cliff boundaries. Third, as a method: a cheap, training-data-only estimate of each molecule's "cliff exposure" — the largest measured potency change among its structurally near neighbors — can gate the per-example radius and resolve the tension without needing cliff labels at test time.
Formally, for a training molecule $x_i$ we compute an exposure score $\kappa(x_i)$ from matched molecular pairs in the training set (pairs with structural similarity at or above MoleculeACE's 0.9 threshold), and set the per-example radius to
$$\rho_i \;=\; \rho_0 \, \exp\!\big(-\beta\, \kappa(x_i)\big),$$
where $\rho_0$ is a base radius and $\beta$ a sharpness-of-adaptation parameter. Examples sitting on cliff edges ($\kappa$ large) are trained close to ordinary empirical risk minimization, preserving their sharp decision boundaries; examples in smooth neighborhoods ($\kappa \approx 0$) get the full flatness penalty. The pair $(\rho_0, \beta)$ is learned by bilevel optimization against a validation cliff error meta-objective, in the spirit of LETS Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius, which learns a single global radius from the squared generalization gap — but where LETS adapts one number over time, CAPS adapts many numbers over chemical space, and targets the metric that actually fails.
What Prior Work Does and Doesn't Cover
Four lines of work constrain this proposal. GraphSAM brings SAM into molecular modeling, but only as a weight-space regularizer for supervised graph transformers, with a heuristic time-decayed radius; it reports ROC-AUC 0.928 on BBBP with GROVER (versus 0.917 for Adam) and never touches cliff-specific error or inverse design Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models. The cliff benchmarks supply the measurement apparatus and the negative result that motivates us: on ACNet's out-of-distribution target split the best deep model (a GCN) collapsed to AUC 0.579, while a plain ECFP-fingerprint MLP reached 0.984 on the in-distribution Large subset Activity Cliff Prediction: Dataset and Benchmark; MoleculeACE additionally found that $\text{RMSE}_{\text{cliff}}$ and overall RMSE correlate at only $r = 0.81$ on average, leaving substantial cliff-specific variance for a method to attack Exposing the Limitations of Molecular Machine Learning with Activity Cliffs. The radius-sensitivity literature supplies the mechanism and the cautionary results: the Goldilocks boundaries of $\rho$ and the destabilization of multi-step inner search Lookbehind-SAM: k steps back, 1 step forward, the cliff-edge landing phenomenon where SAM iterates settle beside steep loss cliffs When Do Flat Minima Optimizers Work?, and the demonstration in DISAM that radius calibration must change when gradients from heterogeneous sources are mixed — there, domains in vision; here, we claim, chemical neighborhoods Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts. Finally, BOLD-SAM shows what goes wrong when the perturbation geometry ignores the discreteness of the underlying object: Euclidean balls on binary weights produce a vanishing perturbation effect and metric distortion, remedied only by moving to native Hamming balls Sharpness-Aware Minimization Directly on the Boolean Hypercube. That paper's diagnosis is our motivation for a mechanistic probe: we will measure how often a $\rho$-ball weight perturbation actually changes the model's predicted ranking within a matched pair, since a perturbation that never flips any structural decision regularizes nothing — a candidate explanation for the finding that SAM, SWA, and weight-averaged SAM all failed to beat standard training on the OGB-Proteins biological graph benchmark (no reported result found on Lacuna for the size of that gap) When Do Flat Minima Optimizers Work?.
What no paper in this literature does is make the radius a function of the input's position in chemical space, or evaluate any radius schedule against cliff-specific error. That is the gap this proposal occupies.
Research Program
The program has three stages. Stage one is the diagnostic sweep: train GCN and MPNN predictors on all 30 MoleculeACE targets with SAM at a grid of fixed radii (including $\rho = 0$, i.e. plain SGD), with matched learning-rate tuning per radius using the Polyak-type scheduler, which is specifically designed to decouple step-size calibration from $\rho$ and remains stable at aggressive radii such as $\rho = 0.4$ where cosine annealing degrades Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler. We plot overall RMSE and $\text{RMSE}_{\text{cliff}}$ against $\rho$. The tension hypothesis predicts differently located optima; if instead the two curves peak at the same radius, the tension hypothesis is falsified and the study still delivers the first $\rho$-sensitivity map on cliff-structured data. We will be explicit about the threat here: the $r = 0.81$ coupling between the two metrics means large effects are needed, and we treat stage one as genuinely open rather than a foregone conclusion.
Stage two builds CAPS. The exposure score $\kappa$ is computed once, before training, from matched molecular pairs in the training split — the same transformation formalism ACNet curates at scale (over 400,000 pairs across 190 targets) Activity Cliff Prediction: Dataset and Benchmark. The bilevel learning of $(\rho_0, \beta)$ follows LETS's first-order Hessian approximation, replacing its squared-generalization-gap meta-objective with validation $\text{RMSE}_{\text{cliff}}$; the generalization gap is the wrong target here because it is exactly the quantity that cliff error decouples from. Because per-example radii multiply into the perturbation construction only through the per-example gradient contributions, CAPS adds no forward or backward passes beyond standard SAM, and we will also implement the one-exact-anchor-per-epoch EMA trick from GraphSAM, which exploits the slow directional drift of molecular perturbation vectors and bought up to 155.4% throughput there Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models, to keep the compute story honest.
Stage three tests whether cliff-calibrated predictors make better surrogates for inverse design, which is where chemical space exploration enters as the application regime. LOL-BO, the strongest local latent-space Bayesian optimization method, localizes search with trust regions but applies no worst-case robustness to its surrogate, and its authors candidly report that maximizing penalized logP drove the score to roughly 545 while generating chemically absurd structures such as endless carbon chains, against roughly 26 for the prior best Local Latent Space Bayesian Optimization over Structured Inputs. A surrogate trained to be flat under small structural edits — but correctly sharp at measured cliffs — should be harder to game in this way. We will rerun LOL-BO unchanged except for the surrogate, on penalized logP and DRD3 docking, and score both raw objective values and the chemical sanity of the returned structures.
Proposed Experiments
The primary evaluation runs on MoleculeACE's 30 ChEMBL targets (dataset sizes ranging from 615 molecules for JAK1 to 3,657 for DRD3 Exposing the Limitations of Molecular Machine Learning with Activity Cliffs), reporting overall RMSE and $\text{RMSE}_{\text{cliff}}$ per target and averaged. The secondary evaluation runs on ACNet across its Large, Medium, Small, Few, and out-of-distribution Mix regimes, reporting AUC, with the Mix target split as the decisive generalization test where deep models currently sit at 0.579 AUC Activity Cliff Prediction: Dataset and Benchmark. The tertiary evaluation is the LOL-BO swap on penalized logP and DRD3 docking, reporting oracle-call efficiency (LOL-BO's own reference point is reaching high-performing candidates with 10x fewer oracle calls than competing methods on the TDC leaderboard Local Latent Space Bayesian Optimization over Structured Inputs) plus a human- and rule-scored validity audit of the generated structures, since a higher raw logP score achieved through degenerate molecules is a failure, not a win. All predictor runs use identical encoders, splits, seeds, and an equal hyperparameter-search budget (the fixed-$\rho$ baselines get a grid whose size equals the search space CAPS explores internally).
Baselines
The "do nothing" reference is the same GCN or MPNN backbone trained with Adam, which fixes how much of any gain comes from architecture versus optimizer; GraphSAM's BBBP result — 0.928 ROC-AUC for GraphSAM against 0.917 for Adam on the same GROVER backbone Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models — is the calibration point for how large flatness gains look on molecular benchmarks (about one point). The strongest existing competitor on the cliff question is the ECFP-plus-classical-ML family: an ECFP+MLP reached AUC 0.984 on ACNet's Large subset Activity Cliff Prediction: Dataset and Benchmark, and SVM with ECFP fingerprints was the best-performing method overall on MoleculeACE, though no per-model RMSE values were reported there (no reported result found on Lacuna for individual MoleculeACE RMSE values) Exposing the Limitations of Molecular Machine Learning with Activity Cliffs. We run these with their published featurization, tuned with the same budget as every other method, and treat beating ECFP+SVM on average $\text{RMSE}_{\text{cliff}}$ at matched overall RMSE as the headline success criterion. The strongest optimizer competitor is fixed-$\rho$ SAM with a full radius grid plus GraphSAM's time-decayed schedule, run on identical backbones and data Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models; CAPS beats it if it achieves lower $\text{RMSE}_{\text{cliff}}$ at equal overall RMSE, or better on both. We also run Stochastic Weight Averaging, which obtains flatness essentially for free by averaging SGD iterates and beat standard SAM on graph representation learning tasks in the 42-task comparison When Do Flat Minima Optimizers Work?; if SWA matches CAPS on cliff error, the adaptive-radius machinery is not paying its way. Finally, two ablations isolate the key ingredient: a LETS-style single globally learned radius with the same cliff-aware meta-objective (showing that spatial adaptivity, not just a smarter learned scalar, is what helps), and a random per-example radius scaling matched in distribution (showing the effect is not a generic benefit of per-example noise).
Why not just…?
Why not just use ECFP fingerprints with an SVM? This is the most serious alternative, and the evidence for it is strong: ECFP-based models dominated both cliff benchmarks, including the 0.984 AUC on ACNet's Large subset Activity Cliff Prediction: Dataset and Benchmark. But it falls short in two ways the experiments will make concrete. Fingerprints are non-differentiable bit vectors, so they cannot serve as surrogates inside gradient-based or latent-space inverse design, which is stage three's entire point; and on out-of-distribution target splits every model family struggled, deep ones hardest, so the question of whether any representation can be made to generalize across targets is open. Our honest position: if CAPS does not close the in-distribution gap to ECFP+SVM on $\text{RMSE}_{\text{cliff}}$ while winning on the Mix split and in the design loop, the proposal has failed on its own terms, and we will report it that way.
Why not just use Stochastic Weight Averaging instead of any radius machinery? SWA gets flatness by averaging iterates after ordinary single-pass training, with none of SAM's doubled compute, and it outperformed standard SAM on graph learning tasks When Do Flat Minima Optimizers Work?. But SWA has no radius knob at all — it cannot be told to be flat in one chemical neighborhood and faithful at a cliff edge — and it lands iterates at basin centers without actively minimizing the curvature that erases boundaries. We include SWA as a first-class baseline precisely because it could win on the averaged metrics; the prediction that distinguishes CAPS is the cliff/non-cliff error gap, which uniform averaging has no mechanism to control.
Why not just grid-search a fixed $\rho$, or use GraphSAM's decay schedule or LETS's learned radius? Grid search is exactly what the stage-one sweep provides, so the comparison is built in, and the radius-sensitivity literature already shows fixed $\rho$ fails under heterogeneity: standard SAM underperformed plain training under domain shift until DISAM made the perturbation pressure source-dependent Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts. GraphSAM's schedule is a function of training time, not of the molecule being fit Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models, and LETS's radius is a single global scalar learned from the generalization gap Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius. The global-learned-radius ablation settles this directly: if a single learned scalar with a cliff-aware objective matches CAPS, then spatial adaptivity was the wrong bet and the ablation will say so.
Why not just tighten LOL-BO's trust regions harder? For the design stage, one could argue that smaller trust regions alone would curb the surrogate gaming that produced the absurd ~545-scoring molecules Local Latent Space Bayesian Optimization over Structured Inputs. But the trust region bounds where the optimizer searches, not how wrong the surrogate can be inside that box; the degenerate structures emerged from a locally trusted surrogate whose property surface was exploited. CAPS attacks the surrogate itself. Since we run CAPS inside LOL-BO unchanged, the experiment cleanly separates the two factors: if shrinking trust regions on the original surrogate reproduces the sanity gains, the simpler story wins, and we will check exactly that as a control.
Decisive Checks
Three results would decide the proposal. First, the stage-one sweep must show a measurable divergence between the $\rho$ optimizing overall RMSE and the $\rho$ optimizing $\text{RMSE}_{\text{cliff}}$ — without it, the motivating tension is empirically vacuous and CAPS loses its reason to exist, whatever else it achieves. Second, CAPS must beat the global-learned-radius ablation and random per-example scaling on the cliff/non-cliff error gap, establishing that structure-conditioned adaptivity is the active ingredient rather than generic per-example perturbation noise. Third, on ACNet's out-of-distribution Mix split, CAPS-trained graph models must move meaningfully above the 0.579 AUC ceiling that the best current deep model reaches there Activity Cliff Prediction: Dataset and Benchmark, since fixing generalization across unseen protein targets — not just in-distribution cliff fidelity — is where the practical payoff of a cliff-aware flatness prior ultimately has to land.
References
- Exposing the Limitations of Molecular Machine Learning with Activity Cliffs
- Enhancing Sharpness-Aware Minimization by Learning Perturbation Radius
- Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models
- Activity Cliff Prediction: Dataset and Benchmark
- Lookbehind-SAM: k steps back, 1 step forward
- When Do Flat Minima Optimizers Work?
- Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts
- Sharpness-Aware Minimization Directly on the Boolean Hypercube
- Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
- Local Latent Space Bayesian Optimization over Structured Inputs