Confident and Wrong: Structure Confidence Is a Poor Selection Filter for Protein Function

Eric LoPrete, Pavle JeremićarXiv 2026 · arXiv

TL;DR

High structural confidence scores from models like AlphaFold or Boltz-2 are excellent at predicting if a protein will fold, but they are surprisingly poor at predicting if it will actually work. This study demonstrates that selecting designs based on structural confidence can actually invert success rates, whereas models trained directly on experimental functional data achieve 25-fold higher odds of success in designing selective metal-binding proteins.

What Problem Does This Paper Address?

In the current "Golden Age" of protein design, tools like RFdiffusion and ProteinMPNN allow researchers to generate thousands of plausible-looking protein backbones. However, the bottleneck remains selection: which of these thousands should we actually synthesize in the lab?

The industry standard has been to use "structural confidence" metrics—such as pLDDT (predicted Local Distance Difference Test)—as a filter. The underlying assumption is that if a model is highly confident a sequence will fold into a specific functional shape, that protein is likely to function. The authors at Aether Biomachines challenge this assumption, specifically in the context of designing proteins for rare earth element (REE) separation, a task where subtle differences in binding affinity are the difference between a useful industrial tool and a useless sequence.

Methods

The researchers compared three distinct design strategies to create variants of lanmodulin, a protein that binds lanthanides:

  1. Function-Selected Design: They trained two transformer-based models on in-house experimental data. One model (the "Kinetic Oracle") predicts function ($pK_d$) from sequence and conditions, while the other (the "Sequence Generator") suggests mutations to optimize that function. These were used in a recursive in-silico evolution loop.
  2. Structure-Confidence-Selected Design: A traditional rational-design workflow using molecular dynamics, RFdiffusion, and ProteinMPNN. Crucially, these designs were filtered for high structural confidence (pLDDT) and low RMSD.
  3. Co-Generation-Selected Design: Using DISCO, a model that generates sequence and structure simultaneously around a target ligand.

The team then synthesized and screened 297 designs against terbium (Tb) and dysprosium (Dy) to measure actual binding affinity and selectivity.

Figure 3: Sequence Design. Diagram of how the ki- netic oracle and sequence generator are used in tandem to guide in-silico evolution of protein sequences.

Datasets

  • Training Data: Approximately 2,500 proteins measured in-house, heavily biased toward lanmodulin variants but including a diverse set of other metal-binding proteins.
  • Evaluation Set: 297 novel designs across the three strategies.
  • Baseline: 149 natural lanmodulin orthologs used to determine if the designs exceeded the "natural" range of functional diversity.

Key Insights and Ideas

  • Confidence is a Plausibility Signal, Not a Functional One: A model can be 100% "confident" that a protein will fold into a specific shape, but that shape might lack the electronic or dynamic properties required for chemistry (like binding a specific metal ion).
  • The Selection Inversion: In this study, the "Structure-Confidence" panel had the highest confidence scores and the best expression rate (97%), yet it had the lowest functional success. It essentially optimized for "looking like a protein" while ignoring the actual design objective.
  • Direct Supervision Wins: Models that learn the mapping from sequence directly to measured function ($pK_d$) outperformed structural filters across every metric. They don't need to "explain" the structure to predict the outcome accurately.

Key Results

The quantitative gap between function-guided and structure-guided design was massive:

  • Binding Success: 83.3% of the Function-Selected designs bound their target, compared to only 16.7% of the Structure-Confidence-Selected designs.
  • Odds Ratio: The odds of a design working were 25 times higher when using function-based selection rather than structural confidence.
  • Predictive Power: As shown in Figure 4, no structural confidence metric (pTM, ipLDDT, etc.) consistently separated binders from non-binders across all panels. In contrast, the function-trained model's predicted $pK_d$ achieved a high ROC AUC (0.79 to 0.92) in every group.
  • Selectivity: The function-guided designs didn't just bind; they were better than nature. They outranked randomly drawn natural orthologs in Tb/Dy selectivity 84% of the time.
Figure 5: Selection on structural confidence produced the fewest working proteins. Left: the fate of every design screened, as a share of that panel’s total. The Structure-Confidence-Selected panel expressed as well as any other and then failed at the binding step, leaving 81% of its designs expressed but inactive; the Co-Generation-Selected panel instead lost 27% of its designs to expression failure. Right: binding rate among expressed designs with 95% Wilson intervals; counts are 80/96, 47/76 and 15/90.

Limitations and Future Work

The authors acknowledge two primary bounds on their findings:

  1. In-Domain Constraint: The function model was trained and tested on the same protein family (lanmodulin). It remains to be seen how well these function-guided models transfer to entirely unrelated protein folds where experimental data is sparse.
  2. Generative Confounding: Because the three panels used different generative methods (e.g., transformer-based evolution vs. RFdiffusion), some of the performance gap might stem from the generation process itself, not just the selection filter.

The takeaway for the field is clear: if you want to design function, you must measure function. Structural models are a powerful starting point, but they cannot yet replace the "ground truth" of experimental feedback loops.

Summary updated

Abstract

Structure-guided protein design can generate plausible candidates, but structural confidence does not predict function. We trained a function-guided model on measured metal-binding affinity with protein sequence, ligand identity, and assay conditions, and used it to design rare earth element (REE) binding variants of lanmodulin, a lanthanide-binding protein with four EF-hand domains. We screened 297 designs from three strategies against terbium and dysprosium, adjacent lanthanides among the hardest REE pairs to separate. Selection on structural confidence inverted the outcome it was meant to improve. The Structure-Confidence-Selected panel carried the highest pTM, complex ipLDDT, and composite confidence scores, and expressed at 97%, yet only 16.7% of its designs bound terbium (95% CI 10.4 to 25.7), against 83.3% (74.6 to 89.5) for the function-guided panel, whose confidence scores were the lowest. The odds of binding differed 25-fold between the two panels (Fisher exact $p = 1 \times 10^{-20}$). No structural-confidence metric separated binders from non-binders consistently across panels, whereas the function-trained model's predicted pKd gave the highest ROC AUC in every panel, 0.79 to 0.92, and ranked measured affinity among function-guided binders at Spearman $ρ= 0.98$. On the design objective, Tb/Dy selectivity, the function-guided designs also exceeded the natural lanmodulin family, outranking a randomly drawn ortholog 84% of the time across 149 orthologs. Structural confidence is a plausibility signal, not a selection criterion for function; direct supervision on measured function is what ranks designs.
BibTeX
@misc{loprete2026confident,
  author = {Eric LoPrete and Pavle Jeremić},
  title = {{Confident and Wrong: Structure Confidence Is a Poor Selection Filter for Protein Function}},
  year = {2026},
  eprint = {2611.17391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17391},
}
Figure