The Leakage Horizon: Auditing Opaque Co-Folding Models for Orphan Receptor Deorphanization

Orphan receptors are cell-surface proteins whose structures we can often solve but whose natural activating molecules remain unknown; "deorphanizing" them — finding those molecules — is one of the highest-value, least-solved problems in chemical biology. A new wave of AI co-folding models (AlphaFold3 and its relatives) that jointly predict a receptor's shape and where a small molecule binds could, in principle, screen billions of compounds against orphan receptors in hours instead of the six CPU-years that physics-based docking of 235 million compounds required in a recent successful deorphanization of the receptor GPR139 (Ultra-large virtual screening unveils potent agonists of the neuromodulatory orphan receptor GPR139). But there is a credibility problem: the same study found that AlphaFold3 handled GPR139 well yet collapsed on other orphan GPCRs absent from its training data, and because its training corpus is a trade secret, nobody could tell whether the GPR139 success came from biophysical reasoning or from memorizing related, well-studied receptors. Exactly this ambiguity — high benchmark scores that cannot be attributed to either memorization or generalization — has been diagnosed and quantified for language models, where saturated benchmarks, leaked test sets, and opaque corpora together make published scores uninterpretable (When Do Benchmark Scores Predict Deployment? A Validity Framework for Capability Claims).

This proposal brings that measurement problem to structural biology. The core claim: for an opaque protein–ligand model, we can behaviorally estimate its contamination exposure — the degree to which its predictions on a given receptor depend on having seen similar receptors during training — without any access to its training data, by measuring how its screening and pose-prediction accuracy decays along a graded ladder of receptor novelty, and by how much that accuracy wobbles under "structural paraphrase" (pocket perturbations that preserve the chemistry but destroy memorizable identity). The deliverable is a per-model leakage horizon: a similarity threshold below which the model's predictions are statistically indistinguishable from template recall. That number is what a practitioner needs before trusting a fast learned screener on a never-studied orphan receptor, and nothing in the literature provides it.

Core Claim

Three failure patterns recur across the supplied literature. First, benchmark saturation: deep models hit near-ceiling scores on standard virtual screening benchmarks — a ligand-only convolutional network, blinded to the receptor entirely, "achieved an AUC of 0.98 on the DUD-E dataset—nearly identical to the full receptor-ligand model" (Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structure-based virtual screening) — yet drop to "0.50 (random chance)" when decoy bias is removed. Second, contamination through the protein side rather than the text side: proteochemometric models barely use their protein inputs at all, since shuffling protein embeddings left AUROC nearly unchanged while ligand shuffling collapsed performance (All That Glitters Is Not Gold: Importance of Rigorous Evaluation of Proteochemometric Models). Third, opacity: the audits that do exist for language models — masked-distractor guessing, where GPT-4 fills in non-logical multiple-choice distractors at 57% exact match versus near 0% for uncontaminated open models (Investigating Data Contamination in Modern Benchmarks for Large Language Models) — are string-based and cannot speak to 3D coordinates or chemical graphs (NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark).

The claim is that a molecular analog of these audits is both feasible and decisive. We treat the public structural-bioactivity databases that plausibly approximate any model's training mixture (PDBbind, ChEMBL) as a proxy for the opaque corpus, order evaluation targets by their distance from that proxy, and measure each model's performance as a smooth function of that distance. If accuracy decays smoothly with receptor similarity, the model interpolates over the protein manifold — soft contamination in the sense of When Do Benchmark Scores Predict Deployment?, whose ARC analysis showed o3 scoring 75.7% on ARC-AGI-1 but collapsing to 3–4% on structurally novel ARC-AGI-2 while rankings stayed stable. If accuracy drops off a cliff at some similarity level, that cliff is the leakage horizon. The same logic that detects contamination via events after a training cutoff in language models (Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?) becomes, here, a homology cutoff: receptors whose ligands were characterized after a model's release date are the cleanest test cases, because no version of the model could have seen the answer.

Research Program

The program has three components: a graded benchmark, a paraphrase-robustness metric, and a masked-ligand probe.

The benchmark, which we call the orphan ladder, orders receptor targets into four tiers by distance from the public data proxy. Tier 0 contains well-annotated targets of the DUD-E and ChEMBL-LR kind, where every model should perform and where we verify saturation. Tier 1 contains recently deorphanized receptors — GPR139 is the anchor, with its solved 3.2 Å cryo-EM complex, its five experimentally confirmed agonists (160 nM to 3.6 µM), and its 50 nM optimized lead (Ultra-large virtual screening unveils potent agonists of the neuromodulatory orphan receptor GPR139) — evaluated only with models whose training snapshots predate the ligand disclosures, so the answers are provably unseen. Tier 2 contains receptors with structurally documented but disputed endogenous ligands: GPR161, where cholesterol occupies the pocket but the physiological activator is hypothesized to be a specific oxysterol (GPR161 structure uncovers the redundant role of sterol-regulated ciliary cAMP signaling in the Hedgehog pathway), and GluD2, which binds D-serine and glycine structurally (Ionotropic glutamate-like receptor δ2 binds d -serine and glycine) yet with affinity so low the channel does not open without constitutively active mutations (The low binding affinity of D-serine at the ionotropic glutamate receptor GluD2 can be attributed to the hinge region) — a tier that specifically punishes binding-only shortcuts that ignore activation. Tier 3 contains true orphans with no agreed ligand at all: GPR21 (Cryo‐EM structure of orphan G protein‐coupled receptor GPR21), the cancer-associated olfactory receptor OR2T6 (The Olfactory Receptor Family 2, Subfamily T, Member 6 (OR2T6) Is Involved in Breast Cancer Progression via Initiating Epithelial-Mesenchymal Transition and MAPK/ERK Pathway), and B7-H3, whose receptor partner is unknown in both directions (B7-H3, a checkpoint molecule, as a target for cancer immunotherapy). Decoy construction for tiers without confirmed actives borrows the ChEMBL-LR discipline — property-matched cross-target actives, active clustering, strict target-family separation (MotifScreen: Generalizing Virtual Screening through Learning Protein-Ligand Interaction Principles) — with the modification that the "decoys" for a true orphan are the confirmed actives of homologous non-orphan receptors, so that a model ranking a homolog's known ligands highly against the orphan is direct evidence of template interpolation.

The second component adapts the Robustness Ratio of When Do Benchmark Scores Predict Deployment? — variance under paraphrase divided by variance under random seeds, where values above 3.0 flagged fragile pattern-matching in language models — to structures. We define

$$RR_{\text{struct}} = \frac{\sigma^2_{\text{paraphrase}}}{\sigma^2_{\text{seed}}},$$

where $\sigma^2_{\text{seed}}$ is the variance of a model's pose or docking score across docking-seed randomization, and $\sigma^2_{\text{paraphrase}}$ is the variance across pocket paraphrases: in-silico mutations outside the binding site, compensatory hinge-region substitutions in the GluD2 spirit, and binding-motif transplants between homologs, each validated to leave physics-based docking scores roughly unchanged. A model that has memorized a specific receptor template should swing wildly under paraphrase while a seed change does nothing; a model using pocket chemistry should be as stable under paraphrase as under reseeding. This is the molecular counterpart of paraphrase-sensitivity testing that revealed a 40x difference between two models with identical 100% benchmark scores (When Do Benchmark Scores Predict Deployment?).

The third component is a masked-ligand probe, the structural analog of TS-Guessing (Investigating Data Contamination in Modern Benchmarks for Large Language Models). We mask scaffold fragments of known ligands and ask the model to reconstruct them twice: once conditioned on the true pocket, once conditioned on a scrambled or unrelated pocket. The gap between conditioned and unconditioned reconstruction isolates how much the model actually reads the receptor — the quantity that permutation tests showed to be nearly zero for proteochemometric models on kinases (All That Glitters Is Not Gold) — from how much it merely exploits a prior over drug-like chemical space. This matters because that prior, unlike pocket reasoning, transfers uselessly to orphan receptors.

Throughout, the emphasis differs from prior evaluation work in this space: Chen et al. diagnosed decoy bias in classical CNNs (Hidden bias in the DUD-E dataset), MotifScreen built a leakage-resistant benchmark and a model that survives it, and Paul et al. built a validity framework for language and code benchmarks. None of them estimates the exposure of an already-trained, opaque model on a target family where no evaluation labels exist — which is precisely the deorphanization setting, and the setting where retraining on clean splits is impossible because the model is closed.

Baselines

The naive reference point is "do nothing": report co-folding models' scores on the saturated standard benchmarks, exactly as current practice does. Concretely this means evaluating AlphaFold3-class and open co-folding models on PDBbind/DUD-E-style splits and on static receptor multiple-choice suites. This baseline is not weak; it is the incumbent, and it is the source of the illusion: on DUD-E a receptor-blind ligand-only CNN reaches AUC 0.98 (Hidden bias in the DUD-E dataset), and on static biosecurity MCQs seven of eight models cluster in a 5-point band while nearly all models score at or above 80% on no-code biology tasks (Same Biology, Different Scores: Quantifying the Tool-Use Confound in Agentic Biology Evaluation). Beating this baseline means showing that the same models, on our ladder, produce a discriminating curve instead of a flat near-ceiling line.

The strongest existing competitor is MotifScreen, a model that learns protein–ligand interaction motifs and holds a mean AUROC of 0.680 on its own leakage-resistant ChEMBL-LR benchmark, versus 0.576 for SurfDock, 0.541 for AutoDock Vina, and 0.518 for a PDBbind-trained protein-aware random forest (MotifScreen). It is the strongest competitor because it is the only published method that both survives leakage-resistant evaluation and runs at screening speed (~0.03 s per compound). We will run MotifScreen, AutoDock Vina, and any co-folding model we can query under identical conditions on every tier: same decoy pools, same pocket definitions, same inference compute, and the same tuning budget (default hyperparameters plus one grid of equivalent size per model). For MotifScreen specifically, which is trained rather than merely prompted, we will retrain on a dated snapshot of the public proxy corpus so its exposure is known — it then doubles as a calibration point. Beating it means achieving comparable or better tier-1/2 discrimination and delivering the diagnostic it does not provide: a calibrated leakage horizon with known exposure on the open models.

The physics floor is AutoDock Vina (mean AUROC 0.541 on ChEMBL-LR; no reported result found on Lacuna for Vina on any orphan-receptor tier), run with matched decoy pools. Its role is to mark what memorization-free baseline accuracy looks like on each tier; a learned model that falls to the Vina floor at high novelty has demonstrated that its advantage was exposure, not physics. The shortcut detector is the ligand-only fingerprint KNN of Hidden bias in the DUD-E dataset: AUC 0.98 on biased DUD-E, 0.50 on the unbiased Actives-as-Decoys set. It cannot run at all on tier 3 by construction — no known actives to compare against — and that failure is itself a diagnostic any proposed method must clear: a method that "works" on true orphans must explain where its signal comes from when no ligand distribution exists.

The key ablation runs the exposure curve without paraphrase perturbations (ladder only) and the paraphrase ratio without the ladder (single-tier perturbation only). If the ladder alone already identifies interpolation and the paraphrase adds nothing, the paraphrase machinery should be dropped; the ablation is designed so that result would be visible, not argued away.

Why not just…?

Why not just use physics-based docking, which cannot memorize? This is the most serious alternative, and it is partially right: transparent docking demonstrably delivers real deorphanization — the GPR139 campaign docked 235 million compounds and produced a 50 nM agonist (Ultra-large virtual screening unveils potent agonists of the neuromodulatory orphan receptor GPR139) — and on unbiased data, PDBbind-trained deep models were "not superior to the performance of the docking program AutoDock Vina" (Hidden bias in the DUD-E dataset). But Vina scores barely above chance (0.541) on leakage-resistant evaluation, and the GPR139 screen consumed roughly six CPU-years; the entire practical case for learned screeners is throughput (FABind: 6.4 Å mean ligand RMSD at 0.12 seconds per complex, versus DiffDock's 20–80 seconds, per FABind: Fast and Accurate Protein-Ligand Binding; Ligand-Transformer roughly 100x faster than Boltz-1, per Sequence-based virtual screening using transformers). Our proposal does not try to replace docking; it answers the question docking advocates currently cannot: for which receptors is the fast learned model safe to use, and where must you pay for physics? Vina is the floor baseline, and if the audit concludes that no opaque model beats the floor outside its leakage horizon, that is a publishable and useful negative result.

Why not just build another leakage-resistant benchmark with target-disjoint splits? This is the MotifScreen/ChEMBL-LR recipe, and it is the closest prior work. It falls short for two reasons grounded in the file. First, splits control leakage for models you train yourself; they say nothing about an already-trained opaque model, which is the entire problem the GPR139 authors faced when they could not decouple AlphaFold3's success from memorization (Ultra-large virtual screening). Second, clean splits do not rescue interpretation of scores: models collapse on ChEMBL-LR (random forest at AUROC 0.518, KarmaDock's collapse reported without an exact number on Lacuna) yet the ranking of models stays stable, just as ARC scores degraded 95% while rankings held at τ > 0.9 (When Do Benchmark Scores Predict Deployment?) — a single AUROC on a hard tier cannot distinguish "bad at this receptor" from "never saw anything like it," while a fitted decay curve can. Our ladder and paraphrase ratio are diagnostics on top of a benchmark, not merely a harder benchmark; the ablation above is exactly the experiment that would show whether the diagnostic layer earns its keep, and we will run it.

Why not just audit the training corpus directly with string or retrieval matching? Direct corpus search works only when the corpus is public: retrieval and n-gram overlap exposed TruthfulQA overlap with C4 for open models (Investigating Data Contamination in Modern Benchmarks for Large Language Models), and the LM Contamination Index registry depends on disclosure (NLP Evaluation in trouble). AlphaFold3's training mixture is proprietary, and even with access, 3D coordinates and chemical graphs leak through similarity rather than verbatim strings — a receptor 40% identical to a training template is contaminated in the practical sense without sharing a single token. Behavioral probing is the only route for closed models, which is why we calibrate the method on open models (MotifScreen retrained on dated snapshots; the Ligand-Transformer family) where ground-truth exposure is known, then apply it to the opaque ones. The honest limitation — behavioral audits are non-exhaustive, and a model can fail recall tests while retaining distorted parametric structure — is inherited from the language-model literature and stated up front; our calibration experiments bound how often the curve mis-estimates exposure on models where we know the answer.

Proposed Experiments and Decisive Checks

Experiment one establishes the saturation illusion and the ladder. We evaluate all models on tier 0 (DUD-E and ChEMBL-LR protocols, where reported numbers exist for anchoring: MotifScreen 0.680, Vina 0.541, ligand-only 0.98 on DUD-E) and on tiers 1–3, then fit each model's accuracy-versus-similarity curve, with similarity defined as maximum sequence identity and pocket-template similarity to the dated public proxy corpus. The decisive readout is the shape: a cliff localizes hard memorization; a smooth decay indicates interpolation over the protein manifold; flat-above-floor performance at zero similarity would refute the memorization hypothesis for that model, as the Ligand-Transformer's prospective 58% hit rate on a mutant EGFR with a 1.185 nM compound hints is possible for at least some architectures (Sequence-based virtual screening using transformers).

Experiment two is the temporal check on GPR139. Models with training snapshots predating the 2025 ligand disclosures are asked to predict the binding pose of the confirmed agonists against the solved 3.2 Å cryo-EM structure and to rank the five confirmed agonists within property-matched decoy pools (their Tanimoto similarity to known ligands is below 0.30, limiting ligand-side shortcuts). Post-release versions of the same model families are asked the identical questions; a performance gap between snapshots is contamination by construction, the molecular analog of the post-cutoff reasoning collapse in language models (Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?) and of the unearned +7 to +17 point jumps documented for post-November-2023 releases (When Do Benchmark Scores Predict Deployment?). No reported result found on Lacuna for any co-folding model's pose accuracy on the GPR139–Compound 1.1 complex, which is why this experiment is worth running rather than citing.

Experiment three measures $RR_{\text{struct}}$ across tiers and models, reporting the paraphrase-to-seed variance ratio with the thresholds (below 1.5 robust, above 3.0 fragile) transplanted from the language-model calibration as initial, explicitly provisional cutoffs to be re-estimated on the open-model calibration set. Experiment four runs the masked-ligand probe to quantify pocket-conditioning, expecting — following the kinase permutation results (All That Glitters Is Not Gold) — that some widely used models show near-zero conditioning gap, which would explain their tier-3 failure before it is even measured. The final deliverable is a public per-model leakage-horizon table: for each screener, the receptor-similarity threshold at which its advantage over Vina ceases to be statistically significant, with tier-3 orphans classified as screenable or not screenable accordingly. Success is a table a medicinal chemist can act on; failure would be curves too noisy to fit, which itself would show that behavioral auditing does not transfer from text to structure and would sharpen where the opacity problem actually bites.

References

  1. Ultra-large virtual screening unveils potent agonists of the neuromodulatory orphan receptor GPR139
  2. When Do Benchmark Scores Predict Deployment? A Validity Framework for Capability Claims
  3. Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structure-based virtual screening
  4. All That Glitters Is Not Gold: Importance of Rigorous Evaluation of Proteochemometric Models
  5. Investigating Data Contamination in Modern Benchmarks for Large Language Models
  6. NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
  7. Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
  8. GPR161 structure uncovers the redundant role of sterol-regulated ciliary cAMP signaling in the Hedgehog pathway
  9. Ionotropic glutamate-like receptor δ2 binds <scp>d</scp> -serine and glycine
  10. The low binding affinity of D-serine at the ionotropic glutamate receptor GluD2 can be attributed to the hinge region
  11. Cryo‐EM structure of orphan G protein‐coupled receptor GPR21
  12. The Olfactory Receptor Family 2, Subfamily T, Member 6 (OR2T6) Is Involved in Breast Cancer Progression via Initiating Epithelial-Mesenchymal Transition and MAPK/ERK Pathway
  13. B7-H3, a checkpoint molecule, as a target for cancer immunotherapy
  14. MotifScreen: Generalizing Virtual Screening through Learning Protein-Ligand Interaction Principles
  15. Same Biology, Different Scores: Quantifying the Tool-Use Confound in Agentic Biology Evaluation
  16. FABind: Fast and Accurate Protein-Ligand Binding
  17. Sequence-based virtual screening using transformers