Hybrid protein structure models combine two sources of information: known structural templates copied from related proteins in the database, and learned sequence-based reasoning that fills in what the templates cannot cover. These hybrids dominate modern prediction and design, but we currently have no way to tell, for a given prediction, how much of it is biophysical reasoning about the input sequence and how much is quiet copying from a retrieved template. The evidence that this matters is stark: Structural Memorization in AlphaFold shows that AlphaFold 3 keeps predicting the same backbone (TM-score ≥ 0.5) even when 40% of the input sequence is mutated to residues that should disrupt the fold, and its confidence scores track the quality of the database template rather than the physics of the mutated query. Meanwhile, on the design side, EvoStruct drives CDR-loop perplexity down to 1.88 (from 3.27 for the generative baseline RAAD) and sequence recovery up to 0.43, yet wins nothing on physical interface metrics, where the antigen-blind RefineGNN stays competitive on CHIMERA-Bench.
This proposal imports the counterfactual likelihood methodology of bias quantification — minimal pairs, pseudo-log-likelihood scoring, inverted probability pairs — from language models into hybrid protein models. The core move is to treat a hybrid model's sequence-likelihood head as a density estimator and interrogate it under controlled swaps of both the sequence and the conditioning structure. A model that actually reasons about sequence–structure compatibility must respond when either is perturbed; a model that memorizes will not, and the shape of its non-response is a measurable, diagnosable bias signature.
Core Claim
Low perplexity in hybrid models conflates two distinct sources: genuine sequence–structure compatibility and template anchoring, in which the structural encoder simply feeds the model the template's own memorized pattern. These can be separated by counterfactual conditioning, and the separated components predict design failure — bad interfaces, poor contact recovery, conformational collapse — in a way that raw perplexity, internal confidence scores, and geometry-only diagnostics cannot. In particular, we predict that the models with the lowest perplexity on CHIMERA-Bench are also the most template-anchored, which is exactly why their sequence gains do not translate into binding geometry.
The Probe
Let $M$ be a hybrid model that outputs amino-acid probabilities for a sequence $x$ conditioned on a structure $s$ (a template, a backbone, or nothing). Define its pseudo-log-likelihood, following the masked-scoring recipe of CrowS-Pairs, which scores each token from context rather than scoring the demographic token itself to avoid frequency confounds:
$$\mathrm{PLL}(x \mid s) = \sum_i \log P_M(x_i \mid x_{\setminus i}, s).$$
Two counterfactual quantities decompose the perplexity. Faithfulness measures how strongly the model's likelihood responds to an adversarially disruptive mutation $x'$ of the wild-type sequence $x$:
$$F = \mathrm{PLL}(x' \mid s) - \mathrm{PLL}(x \mid s).$$
A biophysical model should show a strongly negative $F$ for core-disrupting mutants; a memorizer shows $F \approx 0$, the sequence-likelihood mirror of AlphaFold 3's geometric invariance to 40% mutation. Anchoring measures how much of the likelihood is driven by the specific template rather than by sequence reasoning:
$$A = \mathrm{PLL}(x' \mid s_{\text{matched}}) - \mathrm{PLL}(x' \mid \tilde{s}),$$
where $\tilde{s}$ is a mismatched or blank conditioning structure. Flat faithfulness combined with strong anchoring is the copying signature; strong faithfulness with weak anchoring is genuine compatibility reasoning. Finally, for multi-state targets we form inverted state pairs, the structural analogue of the coherence skew in OptimismBench, which detects directional bias by asking a model for complementary probabilities and measuring their mismatch: $\mathrm{Skew}(x) = \mathrm{PLL}(x \mid s_{\text{state A}}) - \mathrm{PLL}(x \mid s_{\text{state B}})$. A model that understands conformational energetics should rank states by the sequence's actual preferences; a database-biased model should skew toward whichever state is over-represented in crystallographic deposits.
This is methodologically distinct from the closest existing work. EvoStruct uses perplexity as an optimization target inside a hybrid architecture but never as a diagnostic, and explicitly does not test its own model for template reliance. Structural Memorization in AlphaFold performs the adversarial-mutation stress test but measures only geometric quantities (TM-score, DockQ, pLDDT), never likelihood responses, and offers no remediation signal. How to make the most of your masked language model computes pseudo-perplexity at scale (a 20EL× speedup via a wild-type marginal approximation) but purely in one-dimensional sequence space with no structural counterfactuals. The missing piece all three leave open is exactly what we build: a structure-conditioned, counterfactual likelihood probe.
Research Program
We will build an open probe toolkit implementing $F$, $A$, and state skew for any model exposing sequence logits under structural conditioning, and validate it in three stages. First, calibration: using ProteinGym (2.7 million mutated sequences across 250+ assays), we attach template structures to wild types and check that faithfulness scores track experimentally measured fitness — disruptive mutations with low measured fitness should produce strongly negative $F$. This grounds the probe in physical reality rather than circular model outputs; the supervised reference point is ProteinNPT's Spearman 0.613, and the zero-shot reference is TranceptEVE, the top zero-shot substitution model on ProteinGym (no reported result found on Lacuna for its overall Spearman). Second, diagnosis: we probe the CHIMERA-Bench co-design models — EvoStruct, RAAD, RefineGNN, MEAN — across all three splits, correlating anchoring with fnat, DockQ, and contact-residue recovery, and testing whether the perplexity gains EvoStruct reports (1.88 versus RAAD's 3.27) are anchored rather than faithful. Third, conformational bias: we compute inverted state pairs on the 15 experimentally validated fold-switching proteins from the AlphaFold memorization study and on kinase active/inactive chimeras of the kind constructed by Schwarz et al., who stitch rigid domains from one kinase conformation to flexible elements from another; we test whether state skew follows deposition frequency instead of energetics.
Two sub-studies sharpen the methodological lens. The spatial decomposition computes $F$ and $A$ separately for template-derived framework regions and de novo CDR loops — the exact partition that hybrid loop modeling, from Sphinx to Marks's CDR-H3 work, is built around — testing whether the "perplexity paradox" of Comparing Template-based and Template-free Probing operates at the spatial boundary: in natural text, low perplexity tracks accuracy, but under rigid templates perplexity positively correlates with accuracy and reflects frequency bias. And a markedness test, prompted by Template-Based Probes Are Imperfect Lenses, which showed that explicit demographic markers in text templates distort model outputs toward pretraining frequencies: we measure whether the mere presence of template coordinates suppresses the probability of valid non-template-like substitutions, connecting to the vocabulary collapse (tyrosine/glycine over-prediction) that motivated EvoStruct.
Baselines
The do-nothing reference is current practice: report raw perplexity as the quality signal, as EvoStruct does when it cites PPL 1.88 versus 3.27 as evidence of superiority. We beat this baseline by showing that raw PPL carries little or no signal for fnat and DockQ on CHIMERA-Bench — EvoStruct's own interface results already suggest this — while our decomposition adds significant predictive correlation with interface quality. The naive reference point is the model's own confidence: pLDDT and pTM scores, which the AlphaFold memorization study shows select the most accurate structure only about 25% of the time and track template crystallographic quality rather than sequence fidelity; any usable diagnostic must beat 25% model-selection accuracy on the adversarial sets. The strongest existing competitor is the geometric template-closeness diagnostic implicit in Structural Memorization in AlphaFold: measure TM-score or RMSD between the prediction and the nearest database template, flagging invariance under a mutation ladder (AlphaFold 3 holds TM-score ≥ 0.5 up to 40% mutation). We will run it on exactly our complexes, mutation schedules, and compute budget, and beat it in two ways: higher AUROC at separating legitimately homologous predictions (which should resemble their templates) from erroneous copying (which should not, given the disrupted sequence), and applicability to co-design models that output only sequences conditioned on a fixed backbone, where geometry is an input constant and cannot vary. The key ablation isolates our central ingredient: a plain sequence-only protein language model (ESM-2) scoring $\Delta \log P$ between wild type and mutant, in the style of TranceptEVE, with no structural counterfactuals at all. If this matches our probe's AUROC on mutation detection, the counterfactual conditioning adds nothing for that subtask — and the head-to-head is designed to find that out honestly.
Why not just…?
Why not just read the model's confidence scores? Because they are demonstrably an echo chamber: the adversarial study found AF3's confidence metrics select the correct model only ~25% of the time and correlate with the crystallographic quality of the nearest template, not with whether the network actually modeled the mutated sequence. A diagnostic that shares the failure it is meant to detect is not a diagnostic.
Why not just score sequences with an unconditioned protein language model? This is the most serious alternative, and we treat it as such — TranceptEVE-style zero-shot likelihoods already win ProteinGym substitution benchmarks, and McCarter et al. showed ESM-2 likelihoods rival antibody-specialized models. It may well match our probe on the pure mutation-detection subtask, which is why the head-to-head ablation is central. But a sequence-only model cannot express $\mathrm{PLL}(x \mid s_A)$ versus $\mathrm{PLL}(x \mid s_B)$ at all, so the conformational-skew study is impossible by construction; it cannot localize bias to template versus de novo regions because it has no regions; and Distributional Readout warns that sequence-only likelihood is itself vulnerable to memorization, recalling numeric values at $r = 0.98$ while ranking them across contexts at 52.5% — far below the 92% marginal-sampling baseline. A likelihood channel that can memorize is a shaky foundation for a memorization detector; conditioning it on counterfactual structures is what gives the probe its discriminating power.
Why not just measure geometric closeness to the nearest template? It works when the model outputs coordinates, and it already exposes AlphaFold 3's invariance. But it cannot distinguish legitimate homology from copying — a truly homologous protein should match its template — whereas the likelihood drop under disruptive mutation separates precisely these cases, because a faithful model must downgrade a sequence that no longer fits the geometry. More decisively, it is inapplicable to the co-design setting on CHIMERA-Bench, where the backbone is fixed input and only sequences are generated; there, the likelihood response is the only channel that varies. If the geometric diagnostic matches our AUROC on structure-output models, we will report that plainly and rest the contribution on the design-model and multi-state settings.
Why not just fix the models instead of measuring them? Because no one knows yet which component to fix: the memorization study diagnoses without remediating, and EvoStruct's interface failure has no identified cause. The probe's decomposition — flat faithfulness here, strong anchoring there — is precisely the information a remediation needs, and it doubles as the acceptance test for any fix.
Proposed Experiments and Decisive Checks
Experiment one validates faithfulness on ProteinGym: Spearman correlation between $F$-derived scores and measured fitness across substitution and indel assays, with the indel assays probing whether structural conditioning handles length changes that alignment-based models like EVE cannot. Experiment two runs the full probe on CHIMERA-Bench's 2,922 antibody–antigen complexes across the epitope-group, antigen-fold, and temporal splits; the decisive test is whether anchoring $A$ predicts fnat, DockQ, and contact-residue recovery with a sign and magnitude that raw perplexity does not, and whether high-anchoring models concentrate their failures at antigen-contact residues, as EvoStruct's unexplained contact deficit suggests. Experiment three applies the mutation ladder (5–40% substitutions, up to 10% deletions) to inverse-folding heads on the 200 post-cutoff proteins from the memorization study, predicting likelihood invariance in template-conditioned heads mirroring AF3's geometric invariance, versus sharp drops in template-free heads mirroring ESMFold's sensitivity. Experiment four computes inverted state pairs on the 15 fold-switching proteins and kinase DFG-in/DFG-out chimeras; the decisive check is whether skew sign tracks deposition counts rather than the known energetics of the fold-switchers, and whether the coherence failures echo the within-prompt versus cross-prompt gap Kotawala measured. The proposal succeeds if the decomposition predicts interface failure and conformational collapse where perplexity, confidence, and geometry alone do not; it fails, informatively, if the sequence-only ablation matches the probe everywhere the structural counterfactuals are supposed to matter.
References
- Structural Memorization in AlphaFold: Adversarial Mutations Reveal Template Reliance, Confidence Failures, and Implications for Protein Design
- EvoStruct: Bridging Evolutionary and Structural Priors for Antibody CDR Design via Protein Language Model Adaptation
- CHIMERA-Bench: A Benchmark Dataset for Epitope-Specific Antibody Design
- CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
- OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment
- How to make the most of your masked language model for protein engineering
- ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
- TranceptEVE: Combining Family-specific and Family-agnostic Models of Protein Sequences for Improved Fitness Prediction
- Modeling conformational flexibility of kinases in inactive states
- Sphinx: merging knowledge-based and ab initio approaches to improve protein loop prediction
- Antibody H3 Structure Prediction
- Comparing Template-based and Template-free Language Model Probing
- Template-Based Probes Are Imperfect Lenses for Counterfactual Bias Evaluation in LLMs
- Distributional Readout: A Memorization Regime in Autoregressive Generative Models