Protein allostery — the way a perturbation at one site of a protein mechanically changes the behavior of a distant site — is increasingly predicted by machine learning models rather than by expensive physical simulations. But we do not currently know whether these models predict allosteric signaling or merely recall it. The reason is a contamination problem familiar from language-model evaluation: protein foundation models like ESM-2 and AlphaFold2 were pretrained on the UniProt sequence database and the Protein Data Bank up to fixed historical dates, and the benchmarks used to score allosteric and fitness predictors — above all ProteinGym — aggregate experiments published over the past decade, with no chronological firewall between what the model saw during training and what it is tested on. In language models, this exact problem is handled by temporal data isolation: benchmarks such as LiveCodeBench only score a model on problems published after that model's documented training cutoff, so the model cannot have memorized the answers.
This proposal builds AlloBench, the first temporally isolated, continuously refreshed benchmark for allosteric signaling and mutation-effect prediction. Its design question is the mirror image of LiveCodeBench: instead of asking "does the model's score drop on post-cutoff problems?", we ask "how much of the celebrated performance of dynamics-aware protein models survives when the assay, the structure, and the literature are guaranteed post-cutoff?" The stakes are concrete. TriFit, a supervised model that fuses sequence, static structure, and harmonic dynamics, reports an AUROC of 0.897 on ProteinGym substitutions — but its component foundation models could plausibly have already "seen" much of that signal through pretraining, and TriFit itself enforces no date-based split. If a large share of that 0.897 evaporates under temporal isolation, the field's evaluation practice is systematically overstating what allosteric ML has learned; if it barely drops, we gain the first honest evidence that these models generalize. Either result is worth having.
Core Claim
The central claim is twofold. First, methodological: chronological cutoffs — the deposition date of a deep mutational scanning (DMS) assay, the release date of a cryo-EM structure, the publication date of the paper reporting an allosteric pathway — are a causal firewall for biomolecular ML in exactly the way calendar timestamps are for code benchmarks, and they can be enforced programmatically because every relevant database record (PDB entries, UniProt versions, DMS assay depositions, publication preprints) carries a verifiable timestamp. Second, empirical: the apparent contribution of physics-derived features to mutation-effect prediction is currently unquantified under contamination-controlled conditions, and a temporally isolated benchmark is the only clean instrument for measuring it. The proposal deliberately treats temporal data isolation as its core mechanism and allostery as the application regime where the mechanism bites hardest, because allosteric conduits — hydrogen-bonded junctions, loop switches, soft elastic modes — are precisely the kind of conserved structural motif that sequence-identity clustering fails to control for, since two proteins can share a mechanical conduit while sharing almost no sequence.
A second, deeper claim motivates the benchmark's design. Test of Time showed that in language benchmarks, surface-level transformations "launder" the contamination signal: verbatim cloze questions showed a post-cutoff drop of 3.83 percentage points for GPT-4o-mini, while LLM-rephrased questions from the same arXiv papers showed no decay, and mechanistic influence-tracing hit rates collapsed from 77.5% to 17.5%. Biomolecular ML applies such transformations by construction — SE(3)-equivariant architectures like AlloGen's interface transformer and permutation-invariant graph networks like AlloPool consume rotated, translated, re-indexed representations of the same underlying structure. Whether geometric invariance launders structural memorization in the same way phrasing changes launder text memorization is completely unstudied, and AlloBench is designed to measure it.
Research Program
The program has three components: constructing the benchmark, running the temporal audit, and stress-testing the laundering channel.
Construction. AlloBench ingests DMS assays, experimentally determined allosteric structures, and literature-reported allosteric pathway annotations continuously, tagging every item with its earliest public timestamp (PDB deposition date, preprint date, or journal date). Each evaluated model is paired with a cutoff $T$ derived from the documented training window of its foundation components (for a TriFit-style model: ESM-2's and AlphaFold2's training cutoffs, plus the supervised training data cutoff). The model is scored only on assays deposited and structures released strictly after $T$. Because assays arrive continuously, the benchmark resists the saturation and leakage that fixed datasets suffer — the same design philosophy as LiveCodeBench, which scrapes LeetCode, AtCoder, and Codeforces problems with release timestamps and scored over 50 LLMs on 612 problems published between May 2023 and August 2024. The protein analogue of LiveCodeBench's problem feed is the stream of new DMS assays, new cryo-EM/NMR allosteric structures, and new mutational scans of allosteric regulators: enough volume in principle to sustain evaluation, since ProteinGym already catalogs 217 substitution assays covering roughly 696,000 mutations, and new assays accumulate yearly.
Tasks are chosen to probe the allosteric hierarchy specifically, not just generic fitness: (i) predicting the fitness effect of mutations at residues distant from the active site, where a correct answer requires modeling the mechanical conduit rather than local chemistry; (ii) ranking residues by allosteric importance against experimentally identified relays — the central-coupler glutamine in clamp loaders, the αC-β4 loop in PKA-C, the helix-hairpin-helix pull-switch in DNA-PK; and (iii) predicting differential state effects, where the same mutation has different consequences in apo versus holo states.
The temporal audit. We re-train and re-evaluate the strongest dynamics-aware predictor — a TriFit-style architecture — under the cutoff, using only pre-$T$ assays for supervised training, and measure AUROC on post-$T$ assays versus the standard ProteinGym protocol. The decisive quantity is the temporal generalization gap: the difference between same-distribution performance and post-cutoff performance. LiveCodeBench documents what this looks like in code: DeepSeek-Coder-33B trails GPT-4 by only 4.3 points on the static HumanEval+ benchmark but by 16.2 points (a 69% relative gap) on temporally isolated LiveCodeBench problems. We want the protein analogue of that 16.2-point number. We also audit zero-shot foundation models (ESM-2/ESM3-class) the same way, which resolves whether their apparent allosteric competence is retrieval.
The laundering stress test. For a sample of benchmark proteins with documented pretraining exposure (structures deposited well before the foundation-model cutoff), we create transformed variants — sequence-shuffled linkers, engineered point mutations away from the conduit, rotated/re-indexed structures — and compare model confidence and influence-traceable reliance on the original versus transformed inputs, mirroring the Test of Time cloze-versus-transformed design. The open question is whether a model can still exploit a memorized allosteric pathway when the surface representation is transformed, and whether that exploitation is detectable at all by surface-level contamination probes.
Baselines
The "do nothing" reference point is the standard ProteinGym evaluation protocol as used by TriFit and the original ProteinGym benchmark study: all assays pooled, no date split, random or family-level splits at most. On that protocol, TriFit reports an AUROC of 0.897 on the substitution benchmark (with a seed standard deviation of 0.0002), its strongest supervised competitor Kermut reports 0.864, ProteinNPT reports 0.844, and zero-shot ESM3 reports 0.769; the ProteinGym study itself reports a Spearman correlation of 0.613 for supervised ProteinNPT. This is the strongest competitor on this question because it is the protocol that produced every celebrated number in the field — including TriFit's headline claim that dynamics features add +2.2 AUROC points. Our comparison runs the identical TriFit-style architecture, identical compute, and identical hyperparameter search budget under two protocols: the standard pooled protocol (reproducing their number as a sanity check) and the temporally isolated protocol (same model, same tuning budget, training data filtered to pre-$T$ assays only). Beating the baseline here is not about exceeding 0.897 — it is about quantifying how much of it survives isolation. Concretely, the benchmark succeeds if it produces a reliable estimate of the temporal generalization gap with confidence intervals across assay families, and the methodological claim succeeds if a model trained only on pre-$T$ data shows a materially smaller gap than the same architecture trained on pooled data, demonstrating that the pooled protocol inflates estimates.
The strongest existing method baseline is zero-shot ESM3 (and ESM-2) scoring, which sidesteps supervised-training contamination because it requires no assay labels — reporting 0.769 AUROC zero-shot on ProteinGym substitutions per TriFit's evaluation. It will be run under the same temporal cutoff (structures and sequences post-$T$ only) with no tuning, since it has no tunable supervised head. The key ablation isolates the proposal's central ingredient — dynamics features — by evaluating the TriFit-style model with and without its GNM-derived flexibility, mode-shape, and cross-correlation inputs, both under the pooled protocol and under temporal isolation. TriFit's own ablation found that adding dynamics to sequence-plus-structure gave +2.2 AUROC points while adding structure to sequence-only gave just +0.2. Our ablation asks the sharper question: does that +2.2 survive the cutoff, or does part of it reflect the foundation models' prior exposure to the assayed proteins' structures and homologs? A third reference point is physics-only prediction: Propensity Optimised Paths, which computes bond-to-bond signal propagation on energy-weighted atomistic graphs with no learned parameters at all (Wu, Yaliraki, and Barahona). Because it has no training set, it is contamination-free by construction and serves as the floor that any learned model must beat to justify its existence under isolation.
Why not just…?
Why not just use sequence-identity clustering splits? This is the field's default answer to contamination, and it genuinely helps — but the file gives two reasons it is insufficient here. First, Test of Time shows models memorize underlying logic, not surface strings, so surface-similarity filters miss the exposure; in proteins the analogue is mechanical rather than sequence similarity. Second, and decisively for allostery, conserved allosteric conduits persist across families that share almost no sequence: Subramanian et al. found a single invariant hydrogen-bonded glutamine junction acting as the central coupler in AAA+ clamp loaders, tolerating widespread mutation everywhere else. A 20–30% identity split can place two proteins with the same mechanical conduit on opposite sides of the split, and the model legitimately "knows" the conduit from pretraining. We will run the clustering split as a baseline and report whether its temporal gap differs from the date-based split's — if clustering fully accounts for the gap, the simpler method wins and we will say so.
Why not just use physics-based simulation and skip ML entirely? Time-Resolved Force Distribution Analysis and Steered MD trace stress propagation with no training data, hence no contamination by definition. The file itself rules this out as the only answer on throughput grounds: all-atom approaches are too expensive for high-throughput mutational screening, and POPs operates on single static structures, missing ensembles and state transitions. The economic case for learned models is made by AlloGen, whose learned interface transformer scores a complex in about 3 milliseconds, screening 100,000 candidates in under a minute on one GPU — throughput no simulation approach approaches. So the honest framing is: physics-only is the contamination-free floor (and we include POPs as a baseline), and the open question is whether learned models add anything over that floor under isolation. That is exactly what AlloBench measures, and if they do not, that is a finding, not a failure of the proposal.
Why not just run post-hoc contamination probes (n-gram overlap, Min-K% Prob)? Chi et al. used Min-K% Prob to verify CausalProbe-2024's freshness, yet models still scored below 70% on the hard subset (Claude 3 Opus) — passing a token-level freshness filter did not mean the model could reason rather than retrieve. More directly, Test of Time showed transformed questions defeat surface probes entirely: the influence-tracing hit rate fell from 77.5% to 17.5% under rephrasing while the models' exploitation of memorized content persisted. In proteins, geometric invariance is a built-in transformation layer, so token- or residue-level probes are likely even weaker here. Our laundering stress test directly tests this: if surface probes report "clean" while date-based isolation still shows a performance gap, the probes are demonstrably insufficient and the temporal protocol is necessary.
Proposed Experiments
The first experiment is the temporal audit itself. Retrain a TriFit-style trimodal model (ESM-2 embeddings, AlphaFold2 neighborhood coordinates, GNM dynamics features, Mixture-of-Experts fusion) on pre-$T$ ProteinGym assays only, where $T$ is set to ESM-2's training cutoff, and evaluate on assays deposited after $T$ across subsequent years. Report AUROC per time bucket (0–1 years post-cutoff, 1–2 years, and so on), mirroring FreshBench's finding that BPC on arXiv text dropped sharply for papers after March 2024 — though in our case the interesting direction is the opposite: whether predictive performance decays as the assay pool gets fresher. The headline metric is the temporal generalization gap relative to the pooled-protocol AUROC of 0.897. A gap near zero supports the field's current claims; a gap resembling LiveCodeBench's 16.2-point relative collapse would indicate the pooled protocol is substantially inflated. Secondary baselines on the same buckets: zero-shot ESM3 (0.769 pooled AUROC reported by TriFit), Kermut (0.864), ProteinNPT (0.844), and POPs as the physics floor, for which no reported result found on Lacuna for a numerical path-prediction score against DMS data — it will be evaluated by our own residue-ranking protocol.
The second experiment is the allosteric-distant-mutation task, scored only on post-cutoff assays of allosterically regulated proteins. Mutations are partitioned by distance from the functional site; the quantity of interest is the performance differential between proximal and distal mutations, compared across models. A model that retrieves local sequence clichés but cannot model conduits should show a large differential; a model that captures mechanical propagation should not. Predicted residue-importance rankings are validated against the experimentally identified relays where the file supplies them: the αC-β4 loop in PKA-C (Olivieri et al.), the glutamine junction in clamp loaders (Subramanian et al.), and the helix-hairpin-helix pull-switch in DNA-PK (Liang and Blundell). Success on ranking is measured by whether known relays fall in the predicted top decile; no reported result found on Lacuna for an existing model's performance on this relay-ranking task, which is part of why the benchmark is needed.
The third experiment is the laundering stress test on a sample of proteins with documented pre-cutoff structural exposure, following the paired cloze/transformed design of Test of Time but with geometric and sequence-level transformations in place of rephrasing. Where feasible we run influence tracing on an open-weight protein language model, acknowledging the cost constraint that Test of Time could only afford deep influence analysis on 40 papers — we will pre-specify a similarly small, pre-registered sample and treat the result as qualitative evidence.
The decisive check tying the proposal together: if the dynamics-feature ablation shows the +2.2 AUROC gain from GNM features largely survives temporal isolation while the foundation-model embeddings' contribution shrinks, the specific claim "physics-derived features carry genuine generalizable signal that pretraining does not" is established under contamination-controlled conditions. If instead the entire performance profile flattens toward the physics-only floor under isolation, the field learns that its evaluation practice has been measuring memory — and AlloBench becomes the standing instrument that keeps that from happening silently again, in the same way LiveCodeBench did for code.
References
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- TriFit: Trimodal Fusion with Protein Dynamics for Mutation Fitness Prediction
- ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
- Prediction of Protein Allosteric Signalling Pathways and Functional Residues Through Paths of Optimised Propensity
- Allosteric communication in DNA polymerase clamp loaders relies on a critical hydrogen-bonded junction
- AlloGen: Conformation-Selective Binder Design with Differential State Scoring
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- The αC-β4 loop controls the allosteric cooperativity between nucleotide and substrate in the catalytic subunit of protein kinase A
- Human DNA-dependent protein kinase activation mechanism