TL;DR
High structural confidence scores from models like AlphaFold or Boltz-2 are excellent at predicting if a protein will fold, but they are surprisingly poor at predicting if it will actually work. This study demonstrates that selecting designs based on structural confidence can actually invert success rates, whereas models trained directly on experimental functional data achieve 25-fold higher odds of success in designing selective metal-binding proteins.
What Problem Does This Paper Address?
In the current "Golden Age" of protein design, tools like RFdiffusion and ProteinMPNN allow researchers to generate thousands of plausible-looking protein backbones. However, the bottleneck remains selection: which of these thousands should we actually synthesize in the lab?
The industry standard has been to use "structural confidence" metrics—such as pLDDT (predicted Local Distance Difference Test)—as a filter. The underlying assumption is that if a model is highly confident a sequence will fold into a specific functional shape, that protein is likely to function. The authors at Aether Biomachines challenge this assumption, specifically in the context of designing proteins for rare earth element (REE) separation, a task where subtle differences in binding affinity are the difference between a useful industrial tool and a useless sequence.
Methods
The researchers compared three distinct design strategies to create variants of lanmodulin, a protein that binds lanthanides:
- Function-Selected Design: They trained two transformer-based models on in-house experimental data. One model (the "Kinetic Oracle") predicts function ($pK_d$) from sequence and conditions, while the other (the "Sequence Generator") suggests mutations to optimize that function. These were used in a recursive in-silico evolution loop.
- Structure-Confidence-Selected Design: A traditional rational-design workflow using molecular dynamics, RFdiffusion, and ProteinMPNN. Crucially, these designs were filtered for high structural confidence (pLDDT) and low RMSD.
- Co-Generation-Selected Design: Using DISCO, a model that generates sequence and structure simultaneously around a target ligand.
The team then synthesized and screened 297 designs against terbium (Tb) and dysprosium (Dy) to measure actual binding affinity and selectivity.
Datasets
- Training Data: Approximately 2,500 proteins measured in-house, heavily biased toward lanmodulin variants but including a diverse set of other metal-binding proteins.
- Evaluation Set: 297 novel designs across the three strategies.
- Baseline: 149 natural lanmodulin orthologs used to determine if the designs exceeded the "natural" range of functional diversity.
Key Insights and Ideas
- Confidence is a Plausibility Signal, Not a Functional One: A model can be 100% "confident" that a protein will fold into a specific shape, but that shape might lack the electronic or dynamic properties required for chemistry (like binding a specific metal ion).
- The Selection Inversion: In this study, the "Structure-Confidence" panel had the highest confidence scores and the best expression rate (97%), yet it had the lowest functional success. It essentially optimized for "looking like a protein" while ignoring the actual design objective.
- Direct Supervision Wins: Models that learn the mapping from sequence directly to measured function ($pK_d$) outperformed structural filters across every metric. They don't need to "explain" the structure to predict the outcome accurately.
Key Results
The quantitative gap between function-guided and structure-guided design was massive:
- Binding Success: 83.3% of the Function-Selected designs bound their target, compared to only 16.7% of the Structure-Confidence-Selected designs.
- Odds Ratio: The odds of a design working were 25 times higher when using function-based selection rather than structural confidence.
- Predictive Power: As shown in Figure 4, no structural confidence metric (pTM, ipLDDT, etc.) consistently separated binders from non-binders across all panels. In contrast, the function-trained model's predicted $pK_d$ achieved a high ROC AUC (0.79 to 0.92) in every group.
- Selectivity: The function-guided designs didn't just bind; they were better than nature. They outranked randomly drawn natural orthologs in Tb/Dy selectivity 84% of the time.
Limitations and Future Work
The authors acknowledge two primary bounds on their findings:
- In-Domain Constraint: The function model was trained and tested on the same protein family (lanmodulin). It remains to be seen how well these function-guided models transfer to entirely unrelated protein folds where experimental data is sparse.
- Generative Confounding: Because the three panels used different generative methods (e.g., transformer-based evolution vs. RFdiffusion), some of the performance gap might stem from the generation process itself, not just the selection filter.
The takeaway for the field is clear: if you want to design function, you must measure function. Structural models are a powerful starting point, but they cannot yet replace the "ground truth" of experimental feedback loops.
Summary updated