Which Tower to Flatten: Sharpness-Aware Training Against Coordinated Graph–Text Attacks on Unseen Molecular Scaffolds

Molecular property predictors increasingly read two inputs at once: the molecule's structure as a graph, and a natural-language description of it such as a literature snippet or an LLM-written caption. The text measurably helps. Under a split that keeps whole chemical families out of training, adding captions raises average classification ROC-AUC on six MoleculeNet tasks from 85.34 to 87.37 (MolCap-Arena). The text is also the easiest input for an adversary to edit, and vision–language work shows that attacks perturbing both inputs together succeed where single-input attacks fail (Coordinated Robustness Evaluation Framework for Vision-Language Models). Nobody has measured how graph–text molecular models behave under such coordinated attacks. Nobody has measured it on the case that matters in discovery either: molecules whose core ring system, or scaffold, never appeared in training.

This note proposes a controlled study of one cheap defense, sharpness-aware minimization (SAM). SAM is an optimizer that trains the weights toward wide, flat regions of the loss surface by first nudging them in the worst direction and then descending from there. The study targets a question the literature currently answers in two contradictory ways: should flatness be imposed on the dominant input's encoder, or on the most fragile one? We measure the answer on unseen scaffolds, not on random splits, where near-duplicate molecules in training can make robustness look better than it is.

Core Claim

We make two testable claims.

The first is about evaluation. For graph–text molecular models, the gap between random-split and scaffold-split performance is larger for robust accuracy than for clean accuracy. Evaluating attacks on random splits therefore overstates robustness beyond what the known clean-accuracy inflation already explains. Two lines of evidence motivate this. MassSpecGym in the Wild shows that near-neighbor molecules in training let models memorize local chemical space. WelQrate and DrugOOD document sharp drops when moving to scaffold splits. A model that has memorized a family's neighborhood may also be locally smooth there and nowhere else.

The second is about defense. Applying SAM's perturbation to the encoder of the input that is most vulnerable to attack improves coordinated-attack robustness on unseen scaffolds more than two alternatives: flattening the dominant encoder, or flattening all weights uniformly. It also does not lose clean scaffold-split accuracy, and it costs roughly two gradient passes per step instead of the roughly tenfold cost of adversarial training.

Two papers take opposite positions on this. Modality-Aware SAM flattens the dominant modality to stop it overshadowing weaker ones, and evaluates only clean accuracy on random splits. VARMAT penalizes input-gradient norms of the weakest modality against attacks, but never touches loss-surface curvature or structural shift. Our experiment is designed so that either answer is informative. If uniform SAM matches the targeted variants, we report that plainly, and the contribution becomes the evaluation protocol and the scaffold-dependence finding.

Research Program

Models. The testbed is a two-tower molecular model: a GIN graph encoder and a biomedical text encoder, fused late and trained end to end with a small prediction head. It is a trainable version of the MolCap-Arena setup, which used frozen embeddings with an SVM. We initialize the towers from MoMu, a contrastively aligned graph–text model, and separately from MolFM, which adds a cross-modal transformer. Comparing the two tells us whether findings depend on late versus attention-based fusion.

A second testbed is the DrugOOD two-tower affinity model. It pairs a GIN for the ligand with BERT for the protein sequence, so the "text" is an amino-acid string. This checks that the conclusions are not specific to captions.

Threat model. Graph edits that change atoms also change the true label, so we do not attack chemistry directly. The graph tower receives bounded perturbations to its node and edge input embeddings. The text tower receives discrete token substitutions optimized with a Gumbel-Softmax relaxation, restricted to meaning-preserving candidates for captions and to conservative residue swaps for proteins. The coordinated attack optimizes both jointly with a gradient-alignment term, following Coordinated Robustness Evaluation Framework for Vision-Language Models and Unveiling the Fragility of Vision-Language Models. We also run independently optimized single-input attacks. These let us test whether the "countervailing" cancellation that Towards Adversarial Attack on Vision-Language Pre-training Models found between image and text perturbations also appears for graphs and text.

Training variants. Every variant uses one objective restricted to a chosen tower. Let $w$ be all model weights, $L$ the training loss, and $P_m$ a mask that keeps only the weights of tower $m$ (graph, text, or both). The objective is

$$\min_{w}\ \max_{\|\epsilon\|\le\rho}\ L(w + P_m\epsilon),$$

where $\epsilon$ is a weight perturbation and $\rho$ its radius. With $P_m$ covering all weights, this is the original SAM of Foret et al..

We set $m$ per minibatch in two ways. "Dominant" picks the tower with the larger Shapley contribution to the loss, as in M-SAM. "Weakest" picks the tower with the larger VARMAT-style probe, meaning feature magnitude times input-gradient norm. We also log per-tower sharpness, defined as the loss increase at the worst $\epsilon$ within radius $\rho$. We then ask whether it predicts which held-out molecules the coordinated attack breaks.

Baselines

The naive reference is standard training with the same encoders, head, and data: empirical risk minimization with AdamW. It is given a matched budget of gradient evaluations, so it gets twice the epochs of SAM, to rule out "SAM just trains longer." On clean data its scaffold-split numbers should be close to published ones. MolCap-Arena reports 87.37 average ROC-AUC with captions and 85.34 graph-only on its six scaffold-split datasets. DrugOOD reports 68.74% OOD AUC for ERM on its scaffold split. Beating the naive reference means higher robust ROC-AUC under the coordinated attack on unseen scaffolds, with clean scaffold-split ROC-AUC no more than one point lower.

The strongest competitor on robustness is multimodal adversarial training. It generates the same joint graph-embedding plus Gumbel-token perturbation for every training batch, the defense Babu et al. propose as future work. It is the empirical upper bound under large attack budgets according to Sharpness-Aware Minimization Alone can Improve Adversarial Robustness. That paper also reports that adversarial training costs about ten times standard training, while SAM on CIFAR-100 gained up to 6.5% robust accuracy and raised clean accuracy from 76.9% to 78.7%. We found no reported result on Lacuna for adversarial training of graph–text molecular models.

We run adversarial training both at its natural cost and at a budget truncated to SAM's wall-clock, with single-step attacks. Beating it outright is not the goal. The targeted-SAM variant counts as competitive if, at equal compute, it matches or exceeds truncated adversarial training on robust ROC-AUC. It should also retain higher clean scaffold-split accuracy than full adversarial training.

The strongest cheap multimodal defense is VARMAT, with its per-modality vulnerability weighting and gradient-norm regularizer. It reports absolute robustness gains of 12.73% on CMU-MOSEI, 22.21% on UR-FUNNY, and 11.19% on AVMNIST, but we found no reported result on Lacuna for molecular data. M-SAM is run as published, with Shapley-selected dominant-tower SAM plus its gradient modulation. AGH-MAT depends on vision-transformer attention layers that late-fusion GIN models lack. We include it only for the MolFM initialization, which has cross-modal attention.

The ablations isolating our key ingredient are uniform SAM, dominant-tower SAM without modulation, and weakest-tower SAM. They share one $\rho$ grid and identical tuning budgets: the same number of configurations, selected on a validation set of scaffolds disjoint from both train and test. Selecting on random-split validation data would reintroduce the leakage we are studying.

Why not just…?

Why not just use multimodal adversarial training? It is the most likely to win on raw robustness, and we treat it as the upper bound rather than a straw man. The case against defaulting to it is cost and the clean-accuracy tax. In hit-finding data with active rates of 0.039% to 0.682% (WelQrate), a small loss in ranking quality can erase screening value. The compute-matched comparison above settles whether that argument holds for molecules. If truncated adversarial training dominates targeted SAM at equal cost, we will say SAM is not the right tool here.

Why not just apply ordinary SAM to the whole model? This is the real threat. Uniform SAM may capture all of the benefit, and choosing a tower may be unnecessary. We do not argue around it. Uniform SAM is the central ablation, and the tower-selection claim stands only if weakest-tower SAM beats it on coordinated robust ROC-AUC over unseen scaffolds, across seeds and on both testbeds. If it does not, the finding is still useful: flatness helps against coordinated attacks on unseen scaffolds, with no targeting needed.

Why not just drop the text, or use a descriptor MLP? Removing the attackable input removes the attack. WelQrate also shows a descriptor-based MLP is a formidable baseline. Against that, MolCap-Arena's scaffold-split gain shows text carries signal a graph-only model lacks. We report graph-only and descriptor-MLP models as the "robust by abstention" reference. A multimodal defense is only worth using if its robust accuracy on unseen scaffolds exceeds the graph-only model's clean accuracy.

Why not just use an out-of-distribution method such as IRM for the scaffold shift? DrugOOD already tested this. On its scaffold split, IRM (67.74%) and DeepCoral (67.83%) fell below ERM (68.74%). We include IRM once on DrugOOD for continuity but do not expect it to matter.

Proposed Experiments

Data. We use the six MolCap-Arena MoleculeNet tasks with Bemis-Murcko scaffold splits, a Tanimoto-cluster split, and a random split, all at identical sizes. We add the DrugOOD scaffold split and one or two WelQrate datasets scored with logAUC and BEDROC, which measure early enrichment when actives are rare. Metrics are clean and robust ROC-AUC (or R² for regression), attack success rate, and training wall-clock.

Evaluation check. The first decisive check tests the evaluation claim. We compute the random-minus-scaffold gap for clean and for coordinated-robust accuracy under every training method. The claim holds if the robust gap exceeds the clean gap consistently. We also stratify test molecules by nearest-training-neighbor Tanimoto similarity. Attack success should rise as similarity falls if local memorization is what provides apparent robustness.

Defense check. The second decisive check is the tower comparison under a strengthened attack: more steps, restarts, and transfer from a separately trained surrogate. This rules out SAM merely obscuring gradients, a failure that would show up as transfer attacks beating white-box ones. We also report whether per-tower sharpness on held-out scaffolds predicts per-molecule attack success. That link, if present, would explain why one tower should be flattened.

Text shortcut check. Finally, following the shortcut warnings in MassSpecGym in the Wild, we train a text-only model and test whether captions name scaffolds or encode dataset-specific phrasing. If they do, text-tower "vulnerability" may partly be shortcut reliance. We then rerun the key comparison on scaffold-scrubbed captions.