Does Pharmacophore Abstraction Really Resist Dataset Bias? A Stress Test for Shortcut Learning and Training-Data Leakage

A pharmacophore is a coarse three-dimensional map of what a molecule needs to bind a protein: where its hydrogen-bond donors, hydrophobic patches, and charged groups must sit in space, ignoring the exact atomic skeleton. Because this description throws away most of the molecule, it is widely believed to be a safe abstraction — a representation so coarse that a model built on it cannot memorize the particular chemicals it was trained on. That belief matters because of a second, quieter problem running through modern machine learning: training data is often unspecified, proprietary, or biased in ways nobody audited, so a model's benchmark scores may reflect hidden data composition rather than genuine learning. Two recent results make the pharmacophore belief directly testable rather than folklore: deep screening models trained on a biased benchmark were shown to score near-perfectly while ignoring the protein entirely, and black-box molecular encoders were shown to leak their private training molecules to an attacker.

This proposal builds the stress test that the pharmacophore literature has skipped. PharmacoNet, which generates pharmacophores from protein pockets with a 3D segmentation network and scores matches with only seven physical parameters, asserts that this coarse-graining prevents overfitting to training-set chemistry — but the blinded controls that exposed shortcut learning elsewhere have never been run on it or on PharmacoForge, its diffusion-based counterpart. Likewise, Molextract proved that 2D molecular graph networks leak their training molecules at an average precision of 49.0%, but no one has attacked a 3D pharmacophore encoder. We will run both audits, and then convert the diagnosis into something forward-looking: a confidence signal over the ensemble of competing pharmacophore alignments, modeled on how lattice-based confidence estimation fixed overconfident black-box speech recognizers.

Core Claim

The claim under test is precise: coarse-graining molecular interactions into pharmacophore features is not automatically an inductive bias against dataset bias and training-data memorization — it must be measured, because the abstraction can either destroy the shortcut signal (immunity) or merely relabel it (a pharmacophore-feature histogram can correlate with analogue clusters just as a 2D fingerprint can). We hypothesize the answer is mixed and granularity-dependent, and that the honest way to find out is to subject pharmacophore pipelines to the exact controls that collapsed atomistic models, plus the extraction attacks that broke 2D encoders, plus a new confidence mechanism that flags when a pharmacophore screen is operating in a shortcut regime.

The central diagnostic is a pocket-faithfulness gap. Let $s(L, P)$ be a screening score for ligand $L$ against pocket $P$, and let $P_{\text{scr}}$ be a pocket from an unrelated target. We define

$$\Delta_{\text{pocket}} = \mathrm{AUC}\big(s(L, P)\big) - \mathrm{AUC}\big(s(L, P_{\text{scr}})\big),$$

the area-under-curve drop when the model is blinded to the real pocket. A pocket-faithful model has large $\Delta_{\text{pocket}}$ on biased benchmarks and retains its performance when decoys are replaced by other targets' genuine actives; a shortcut model has near-zero $\Delta_{\text{pocket}}$ on the biased benchmark and collapses on the unbiased one — exactly the pattern the hidden-bias study demonstrated for atomistic CNNs.

Research Program

Part 1: Blinded controls adapted to pocket-derived pharmacophores. The dummy-atom trick from the DUD-E bias study — replacing the receptor with a single atom — does not transfer directly, because pharmacophore models generate their query from the pocket. We therefore design three pocket-side controls: pocket scrambling ($P_{\text{scr}}$ substitution, per the equation above), pharmacophore shuffling (replacing the generated query with a feature-matched random pharmacophore), and feature-histogram matching (checking whether the distribution of feature types alone — donors, hydrophobes, aromatics — already separates actives from decoys, which would mean the model exploits analogue bias in a new vocabulary). We run these controls on a rebuilt PharmacoNet-style pipeline trained on PDBbind v2020, a PharmacoForge-style diffusion model trained on CrossDocked2020, and the deterministic generator Apo2ph4, across DUD-E, the Actives-as-Decoys set, and LIT-PCBA. Apo2ph4, which produces receptor-based pharmacophores without learned components, serves as the bias-invariant reference point.

Part 2: Extraction attacks on 3D pharmacophore encoders. We retrain pharmacophore encoders (hotspot-prediction embeddings and GVP diffusion encoders) on a simulated proprietary split — 2 million ZINC15 pocket–ligand pairs, mirroring the setup in the Molextract paper — then adapt its adapter-plus-reinforcement-learning attack to reconstruct training pairs from black-box pharmacophore embeddings, including a test of whether its linear-combination heuristic ($f(\hat{G}) \approx \alpha f(R) + (1-\alpha) f(M)$ for training molecules) survives the lossy coarse-graining. Alongside the attack we measure the empirical preimage size: how many distinct molecules in ZINC20 map to the same pharmacophore query. That number sets an information-theoretic ceiling on reconstruction precision and tells us whether abstraction is a genuine privacy shield or just a thinner fingerprint.

Part 3: Alignment-lattice confidence. Pharmacophore matching is not a single answer — the graph-matching step yields an ensemble of competing alignments, and current tools collapse it into one deterministic score. This mirrors speech recognition, where discarding alternative hypotheses caused overconfidence until lattice-based confidence estimation raised confidence-prediction AUC from 0.7496 (word-only baseline) to 0.9231 by reading the whole hypothesis graph with a bidirectional recurrent network. We train a small bidirectional network over the alignment graph to predict screening errors per compound, then evaluate whether its confidence flags the exact targets and compound clusters where Part 1 found shortcuts — turning the audit into a deployable warning signal.

Baselines

The naive reference point is 2D-fingerprint K-nearest-neighbors ligand similarity, the method the hidden-bias study showed performs comparably to 3D CNNs on DUD-E by exploiting analogue bias. It is the strongest competitor for the diagnostic question because it is the shortcut, made explicit: any pocket-aware model that fails to beat it on the unbiased Actives-as-Decoys set is not using pocket information. We run it on identical target sets and libraries, and "beating" it means a pocket-faithfulness gap $\Delta_{\text{pocket}}$ that KNN structurally cannot have (it sees no pocket at all) plus retained enrichment when decoys are other targets' actives.

The strongest existing learned method is PharmacoNet, whose pipeline we rebuild and evaluate under identical data, tuning budget, and compute as our variants; the file reports that it outperformed other docking-free deep models on LIT-PCBA and screened 187 million ZINC20 compounds in 21 hours on a 32-core CPU, but no reported result found on Lacuna for its per-target enrichment factors, so we will re-measure it ourselves rather than compare against unreported numbers. For context on what strong contrastive competitors achieve on LIT-PCBA, SaBAN reports an AUROC of 68.16% (a 10.99% absolute improvement over DrugCLIP) and a BEDROC of 13.35%, and Uni-Clip reports 147% and 218% improvements in enrichment factor at 1% on LIT-PCBA and DUD-E respectively; both will also be run through our blinded controls, since their pre-training corpora (850,000 MMBind pairs, BindingDB, AlphaFold structures) are precisely the unspecified training data whose overlap with evaluation targets is unaudited. For the extraction experiment, Molextract is the strongest competitor, with its reported average precision of 49.0% (76% on AttrMasking models) as the number our 3D attack must be interpreted against — lower precision could mean either a weaker attack or a stronger shield, which is why the preimage measurement in Part 2 is needed to disambiguate. Finally, the key ablation isolates granularity itself: the same scoring architecture run at atomistic resolution versus pharmacophore resolution on the same training data and controls, so that any difference in shortcut resistance is attributable to the abstraction rather than to architecture, data, or tuning.

Why not just…?

Why not just use physics-based docking and skip the training-data problem entirely? Docking has no training distribution to specify, and the hidden-bias study found PDBbind-trained deep models were no more effective than AutoDock Vina on DUD-E, so the temptation is real. But PharmacoNet is 3,956× faster than Vina and screened 187 million compounds in 21 hours where Vina would need an estimated 11 years, and docking scoring fails on flat, hard targets where Uni-Clip found non-covalent inhibitors at 4.17 µM. The question we are asking is whether learned models at ultra-large scale deserve trust, so docking enters as a comparator column in every table, not as the answer; if pharmacophore models fail our stress test, the honest conclusion will indeed be "use docking," and the experiments settle that.

Why not just apply the existing ligand-only control and retrain on curated data? This is the most serious objection, because the hidden-bias paper already supplies the diagnostic, and Part 1 of this proposal is in some sense that control applied to a new model family. Three things make the extension necessary rather than redundant: the dummy-atom control does not transfer to pocket-derived pharmacophores and must be redesigned as pocket scrambling and pharmacophore shuffling; the control says nothing about training-data extraction, which is a distinct failure mode Molextract proved is universal for 2D encoders; and controls diagnose only in retrospect — they cannot flag an unreliable prediction at deployment time, which is what Part 3's confidence signal provides. If a reviewer believes the control alone suffices, Experiment 1's results on pharmacophore models that pass the control yet still leak training pairs in Experiment 2 would directly refute that position.

Why not just use the contrastive foundation models that already report unbiased-benchmark gains? Uni-Clip and SaBAN already evaluate on LIT-PCBA and report large improvements, so one might conclude robustness is solved. The gap is that their pre-training mixtures — MMBind's 850,000 pairs, BindingDB, AlphaFold predictions — are exactly the kind of massive, partially unspecified corpora whose overlap with evaluation targets is never audited; strong LIT-PCBA numbers could reflect retrieval by memorization rather than generalization. Our blinded controls apply to these models unchanged, and this is a case where we treat the alternative as a genuine threat: if Uni-Clip passes every control we devise, the honest conclusion is that pharmacophore abstraction is not the only route to bias resistance, and our contribution narrows to the extraction audit and confidence mechanism.

Why not just generate molecules directly with 3D diffusion models instead of screening with pharmacophores? Because the generative route manufactures its own data problem — PharmacoForge's evaluation found that de novo models DiffSBDD and Pocket2Mol produce molecules with median strain energies of 295.7 and 351.5 kcal/mol (physically impossible conformations), while pharmacophore-queried library hits sit at 0.05 kcal/mol. Generative models trained on CrossDocked2020 also inherit whatever bias that docked-pose corpus carries, which is itself an unspecified-training-data question our Part 2 attack can probe; we include one such model in the audit for exactly this reason.

Proposed Experiments and Decisive Checks

The experimental sequence runs on DUD-E (102 targets, property-matched decoys), the Actives-as-Decoys set (genuine actives of other targets as decoys), LIT-PCBA (15 unbiased HTS-derived targets), PDBbind v2020 and CrossDocked2020 for training, and ZINC15/ZINC20 for the extraction and preimage measurements, all drawn from the benchmarks in the file. Experiment 1 applies the three blinded controls to all model families and reports per-target AUC and enrichment factor at 1%, with pre-registered criteria: a model is declared shortcut-free only if its pocket-scrambled variant falls to within 0.05 of 0.5 AUC on DUD-E and its full-model performance on Actives-as-Decoys stays within 0.03 of its DUD-E performance. Experiment 2 reports attack average precision against the 49.0% Molextract reference, interpreted jointly with the measured preimage sizes; an attack precision within a factor of two of random retrieval over the library counts as leak-resistant, and any model whose pharmacophore queries admit small preimages while still leaking co-occurrence patterns would show that abstraction relabels rather than removes the shortcut. Experiment 3 evaluates the alignment-lattice confidence head by error-prediction AUC on LIT-PCBA, against the naive baseline of using the raw screening score as its own confidence, with success defined as a substantial margin over that naive baseline concentrated on precisely the targets and clusters flagged by Experiment 1. Both outcomes are publishable and decision-relevant: if abstraction holds, we will have quantified, for the first time, a cheap and principled shield against dataset bias and training-data leakage in ultra-large-scale screening; if it fails, we will have exposed masked shortcuts in the very models currently being used to screen hundreds of millions of compounds, and supplied the confidence signal that flags them in production.