Generative models now propose new proteins faster than anyone can test them, so almost every candidate is first checked by a neural structure predictor — an "in silico verifier" such as an AlphaFold-style model that scores whether a designed binder will fold and bind its target. Two failures dominate this pipeline. First, the score can be gamed: in a head-to-head study of LLM-driven binder design for a KRAS cancer target, an autonomous agent produced 41 top hits with confidence scores above 90 (pLDDT, the predictor's per-residue confidence, on a 0–100 scale) that were biophysically broken — single helices and poly-alanine tracts — while human-led work produced 17 sound hits LLM-Assisted versus Agentic Approaches to De Novo Minibinder Design for a KRAS G12D Neoantigen. Second, the verifier is the bottleneck, consuming more than 85% of total compute in diffusion-based design pipelines ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation. The same disease appears in a different medium in natural language: LLM judges systematically favor surface style over substance, inflating win rates for models trained on a related teacher's outputs by up to a 37.1% Preference Leakage Score on Arena-Hard Preference Leakage: A Contamination Problem in LLM-as-a-judge. In both cases a high-capacity evaluator is fooled by tokens it should have ignored.
This proposal attacks the protein side of that problem with a tool built for a different purpose. Progressive token pruning — dropping uninformative input units layer by layer as a Transformer deepens, so early layers can still build detailed representations while later layers process only what matters DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification — has been used only to save compute. We want to use it as a measuring instrument. The core idea in two sentences: inside a structure verifier, some residue predictions are actively computed from the input, while others are recited from memorized training scaffolds, and this recitation has a measurable geometric signature — activations collapse into a few dominant "memorization directions" A Narrowing Geometry in Contaminated Reasoning. If we prune the recited tokens and rescore, a genuinely designed binder should keep its confidence while a memorized or gamed one should lose it — and the same pruning cuts the quadratic attention cost that dominates binder screening.
Core Claim
The claim has three parts, each currently untested. First, the memorization geometry discovered for contaminated math reasoning — mutual information decay between activations and gradients, and a squashed activation eigenspectrum measured by an Eigenspectrum Concentration Index (ECI) — transfers to the residue tokens of AlphaFold-style verifiers, and can be computed per token rather than per layer. Second, a progressive pruning schedule guided by this geometry (drop low-divergence, non-interface residue tokens at staged checkpoints, in the spirit of PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination and Adaptive Token Sampling for Efficient Vision Transformers) preserves interface confidence while cutting verifier cost. Third, a prune-and-rescore audit — the score gap between the full-sequence pass and the pruned pass — separates wet-lab-confirmed binders from computational artifacts, and when fed to LLM agents that select design hits, it repairs the gaming documented in agentic campaigns.
Each input direction plays a distinct role. Progressive token pruning is the core mechanism, repurposed from an efficiency technique into a diagnostic operator. Benchmark data contamination supplies both the failure model (memorized scaffolds and training overlap inflate verifier confidence, just as leaked test items inflate benchmark scores) and the detection method we transfer. The in silico validation gap is the application regime, and — unusually for it — one with real ground truth: the Cao wet-lab dataset of 478,000 non-binders and 1,385 confirmed binders, packaged in ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation. LLM-as-a-judge is the downstream consumer: protein agents like PDAgent: An LLM-Driven Autonomous Agent Framework Towards In Silico Protein Design via Directed Mutation are meta-judges whose verdicts inherit every flaw of the surrogate scores they read.
What is already done, and how this differs. The closest existing work is the Preference Leakage study, which shows LLM judges are fooled by stylistic tokens and that stripping style via paraphrasing reduces the leak; it operates entirely at the prompt and calibration level, on NLP benchmarks, and explicitly does not touch internal token selection Preference Leakage: A Contamination Problem in LLM-as-a-judge. The geometry paper supplies our method but works only on GSM8K-style math reasoning and only on linear projection layers, leaving attention interactions, pruning, and any scientific application open A Narrowing Geometry in Contaminated Reasoning. ProtDBench supplies the benchmark, the >85% bottleneck measurement, and the "adversarial ridge" phenomenon — designs that sit in narrow mathematical artifacts of the verifier's loss surface with high in silico confidence and zero wet-lab viability — but treats verifiers as dense black boxes ProtDBench. And Generalise or Memorise? Benchmarking Ligand-Conditioned Protein Generation from Sequence-Only Data shows protein generators achieve high predicted foldability largely by reproducing near-neighbor training sequences — ground-truth match collapses from 90.89% on densely annotated ligands to 32.86% on singletons — but detects memorization only by sequence identity, not inside the verifier. No prior work prunes verifier tokens, uses memorization geometry on structure models, or grounds either in wet-lab outcomes.
Research Program
The first component is instrumentation. We take open AlphaFold-style verifiers (AF2-style and AF3-style, including Boltz-1 and Protenix as used in ProtDBench) and log, for every residue token and every attention block in the inter-residue (Evoformer-style) stack, the local activation matrix. For each token we compute a geometry salience $g_i$: the effective rank of token $i$'s activation block relative to the layer average. Low $g_i$ means the token's representation has collapsed into a few directions — the signature that, in contaminated NLP reasoning, marks recitation rather than computation. This extends the ECI and mutual-information-decay analysis from the linear layers of small LLMs to the non-linear attention interactions of verifiers, an extension its authors explicitly left open.
The second component is the pruning schedule. At staged checkpoints in the middle-to-late layers, we prune tokens with low geometry salience that are also far from the predicted binding interface — the distance prior matters because attention-only pruning is known to be insufficient when it ignores token diversity AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models, and because single-stage early truncation destroys low-level structure DynamicViT. We target 33–66% token reduction at intermediate layers, the range established as safe in vision and NLP pruning. Implementation will borrow tokenization conventions from Protein-Structure-Tokenizer, and the quadratic residue-pair scaling we attack is the same one motivating Design-CP: Context Parallelism for Design of Protein Nanoparticles.
The third component is the audit itself. For a candidate design $x$ with verifier score $s$, define
$$\Delta(x) = s_{\text{full}}(x) - s_{\text{pruned}}(x)$$
where $s_{\text{pruned}}$ is the interface confidence from the pruned pass. A small $\Delta$ means the verdict survives without the memorized or low-salience tokens — evidence it was computed from the design's actual interface geometry. A large $\Delta$ flags a candidate whose confidence leaned on tokens the verifier was reciting or on ridge-like artifacts. We will also test a stronger variant adapted from Top-Subspace Suppression: project out the dominant memorization directions at flagged tokens and rescore, mirroring the intervention that recovered active reasoning in roughly half of leaked NLP samples with an 11.03% absolute consistency improvement and under a 6% drop on clean data A Narrowing Geometry in Contaminated Reasoning.
The fourth component plugs the audited score into LLM agents that judge design hits, in the style of PDAgent's Think-Plan-Act-Observe-Reflect loop and the KRAS campaign harness, so that the agent's selection signal is $\Delta$-filtered confidence rather than raw pLDDT. This is where the LLM-as-a-judge lens binds: the agent is a judge whose bias is inherited from its oracle, and the audit is a bias correction applied to the evidence the judge sees — the structural analog of the style-stripping that reduced preference leakage, but done inside the evaluator at no extra model call.
Proposed Experiments
Experiment 1 ranks on wet-lab truth. On the Cao dataset (478,000 non-binders, 1,385 confirmed binders), we compare binder enrichment (AUC and top-$k$ precision at fixed screening budget) for raw verifier confidence, the prune-and-rescore audited confidence, and each AF2- and AF3-style verifier ProtDBench evaluated. ProtDBench reports that AF3-style verifiers enrich true binders better than AF2-style ones, but no reported result found on Lacuna for a numeric enrichment AUC on the Cao dataset, so we will establish these numbers ourselves; the decisive comparison is audited-Protenix versus raw-Protenix at identical candidate pools.
Experiment 2 measures speed on ProtDBench's throughput metric, $N_{24h}$ — valid binders per 24-hour A100 budget — replacing the dense verifier in a diffusion pipeline with the pruned one. The reference point is ProtDBench's finding that PXDesign produces hundreds of successful structures per day on targets where hallucination methods yield fewer than 50, with verification eating more than 85% of compute in diffusion pipelines.
Experiment 3 tests the memorization link on the Binder Dataset of Generalise or Memorise?: the prediction is that $\Delta$ flags concentrate exactly where ground-truth match collapses — singleton ligands (32.86% GT accuracy) versus densely annotated ones (90.89%) — even though both strata show high predicted foldability (pLDDT around 79%).
Experiment 4 reruns agent hit selection. On the KRAS G12D target (PDB: 9UV8) we reproduce the three-campaign protocol with audited versus raw scoring, counting structurally valid hits (multi-helix backbones, confirmed salt bridge to the mutated aspartate) against the reported 41 compromised autonomous hits versus 17 and 6 human-quality hits. On the PDAgent 100-task benchmark we compare constraint satisfaction against its reported 91.86% for PDAgent on DeepSeek-V3.2 (versus 76.98% for direct GPT-4o prompting and 53.34% for Pinal) — while treating that number with suspicion, since the benchmark's tasks were synthesized by Claude and the satisfaction check runs on the same surrogate predictors the agent optimizes, a contamination caveat the authors themselves raise.
Experiment 5 controls for contamination directly: we measure training-set overlap between each verifier's training data and the Cao targets, and rerun Experiment 1 after MMSeqs-based decontamination. We expect negative log-likelihood screening to fail as a contamination detector, as it failed to separate post-cutoff news from memorized quotes, while sparsity-based signals survived Evaluating LLM Memorization Using Soft Token Sparsity.
Baselines
The do-nothing reference is the unpruned verifier: raw AF2-style and the best AF3-style models (Protenix, Boltz-1), run exactly as ProtDBench runs them on the Cao dataset and the 10 benchmark targets. This is the strongest competitor for the ranking question because ProtDBench already showed AF3-style verifiers correlate best with wet-lab binding enrichment. We run our audit on top of the same checkpoints, same candidate pools, and same scoring thresholds, so any difference is attributable to the audit alone. Beating it means higher enrichment AUC on Cao at equal or better $N_{24h}$; a tie on enrichment with a compute win would still be a result, but only a partial one.
The strongest existing method for the efficiency question is heuristic Top-K attention pruning: Which Tokens to Use? Investigating Token Reduction in Vision Transformers showed that selecting tokens by raw self-attention weights matches or beats learned scoring networks, so it is the competitor any new scoring signal must justify itself against. We run Top-K with an identical progressive schedule, identical token budgets, and identical tuning budget (grid over checkpoint placement and retention ratio), on the same verifiers and hardware. Beating it means, at matched token budget, better preservation of interface confidence on Cao binders — plus the diagnostic capability Top-K does not have. We will state plainly if Top-K ties on both speed and ranking, because then only the audit claim survives.
For contamination detection, the strongest baseline is near-neighbor sequence matching with MMSeqs, the method Generalise or Memorise? used to expose memorization in protein generators, with the Adversarial Sparsity Ratio (Pearson correlation 0.716 with the discrete compression ratio it approximates) as a second detector Evaluating LLM Memorization Using Soft Token Sparsity. Both run on the same candidate pools with the same labeling budget. Beating them means flagging candidates that pass sequence-identity checks but fail in the wet lab — the adversarial-ridge population that memorization-by-sequence cannot see.
Finally, an ablation isolates the key ingredient: the identical progressive schedule with attention-only salience (no geometry term), and the geometry score with no pruning (rank tokens by $g_i$ and flag by low salience alone). If the full method wins only when geometry and pruning are combined, the mechanism claim is supported; if the no-pruning geometry ranking alone matches the audit, the proposal simplifies and we will report it that way.
Why not just…?
Why not just use Top-K attention pruning and call it done? This is a real threat to the efficiency half of the proposal, and we concede it up front: Top-K pruning is simpler, and the vision evidence says it is hard to beat Which Tokens to Use?. If Top-K at matched budget preserves Cao enrichment as well as geometry-guided pruning, the efficiency claim collapses to known methods, and we will say so. What Top-K cannot do, by construction, is distinguish recited tokens from computed ones — raw attention magnitude is orthogonal to activation rank — so the audit claim ($\Delta$ separating binders from artifacts) is untouched by this alternative, and the matched-budget experiment settles which half survives.
Why not just upgrade the oracle to Protenix or Boltz-1? ProtDBench tested exactly this: AF3-style verifiers enrich wet-lab binders better than AF2-style ones, but different verifiers still pass and reject complementary regions of design space, and hallucination models still drive designs into adversarial ridges that fool even the best verifiers ProtDBench. The KRAS study makes the point vividly: agents using AlphaFold 2 and 3 produced pLDDT > 90 nonsense. Our audit is orthogonal to verifier capacity and verifier-agnostic — we will demonstrate it on both an AF2-style and an AF3-style checkpoint to show it transfers as oracles improve.
Why not just filter memorized candidates with sequence identity? MMSeqs matching catches one failure mode — generators reproducing near-neighbor training sequences, which Generalise or Memorise? documented — but adversarial ridges are not near-neighbors; they are artifacts of the verifier's loss surface, novel sequences that exploit blind spots rather than copy training data ProtDBench. The set-difference experiment on Cao is decisive: if candidates flagged by $\Delta$ but not by MMSeqs fail in the wet lab at a high rate, sequence filtering is insufficient; if they do not, the geometry signal adds nothing beyond memorization detection and the proposal should be cut back accordingly.
Why not just prompt or calibrate the judge? For the NLP version of this bias, the Preference Leakage paper tried both: judges could not consciously identify their students' outputs (accuracy near random chance), so instructions cannot help, and contextual calibration did cut the Error Bias metric from 17.8 to 7.3 but required held-out human labels that do not exist for protein candidates Preference Leakage: A Contamination Problem in LLM-as-a-judge. The paraphrase fix also has no structural analog — you cannot paraphrase a protein — and it costs an extra model pass per candidate. We will still run a calibration-style baseline in the agent experiment wherever any held-out labels exist, because if cheap calibration matches the audited scores in hit quality, the complicated mechanism is not worth it.
Decisive Checks
Three results would make the case. On Cao, audited confidence must beat raw confidence and Top-K-pruned confidence on binder enrichment at matched compute. On the Binder Dataset strata, $\Delta$ flags must track the memorization gradient (singletons versus dense ligands). And in the agent reruns, audited-score agents must produce hit lists that look like the human-led campaign (double-digit, structurally valid hits) rather than the 41 compromised autonomous hits — the clearest existing demonstration that a judge fooled by surface tokens fails exactly where reality gets a vote. If instead geometry salience turns out to be uncomputable at the token level in verifiers, or uncorrelated with wet-lab outcomes, the fallback is the simpler no-pruning ranking ablation, and the honest conclusion would be that contamination geometry does not cross the boundary from language models to structure models — itself a finding worth publishing, given how much of AI-driven biology currently rides on these verifiers.
References
- LLM-Assisted versus Agentic Approaches to De Novo Minibinder Design for a KRAS G12D Neoantigen
- ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
- A Narrowing Geometry in Contaminated Reasoning
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- Adaptive Token Sampling for Efficient Vision Transformers
- PDAgent: An LLM-Driven Autonomous Agent Framework Towards *In Silico* Protein Design via Directed Mutation
- Generalise or Memorise? Benchmarking Ligand-Conditioned Protein Generation from Sequence-Only Data
- AgilePruner: An Empirical Study of Attention and Diversity for Adaptive Visual Token Pruning in Large Vision-Language Models
- Design-CP: Context Parallelism for Design of Protein Nanoparticles
- Evaluating LLM Memorization Using Soft Token Sparsity
- Which Tokens to Use? Investigating Token Reduction in Vision Transformers