Introduction
De novo protein design has transitioned from simple sequence optimization to the generation of entirely novel protein backbones. While generative AI models like RFdiffusion, FrameFlow, and Chroma have demonstrated remarkable potential, the field has lacked a standardized evaluation framework. Disparate datasets, varying protein lengths, and inconsistent metrics have made "apples-to-apples" comparisons nearly impossible. To address this, we introduce Scaffold-Lab, a unified benchmarking framework designed to rigorously assess protein backbone generation methods across designability, novelty, diversity, and efficiency.
The Scaffold-Lab Framework
Scaffold-Lab evaluates models through a comprehensive "refolding pipeline" that tests whether a generated backbone is physically viable. The process follows four critical steps:
- Backbone Generation: The generative model produces a 3D protein backbone.
- Sequence Design: ProteinMPNN is utilized to design an amino acid sequence compatible with the generated backbone.
- Structure Prediction: ESMFold predicts the 3D structure of the newly designed sequence.
- Validation: The predicted structure is compared against the original AI-generated backbone using the self-consistency TM-score (sc-TM). A design is considered successful if the sc-TM exceeds 0.5.
Beyond basic designability, Scaffold-Lab introduces a Novelty Score with a Penalty to ensure models are not rewarded for generating un-foldable "hallucinations." The framework also employs TOPSIS, a multi-criteria decision-making method, to provide an objective ranking of model performance.
Benchmarking Results
Unconditional Generation
Our evaluation of seven state-of-the-art methods revealed a significant "designability watershed." While models perform exceptionally well for short proteins, success rates drop precipitously once proteins exceed 300 residues, with rates falling below 30% for proteins longer than 800 residues.
- Top Performers: FrameFlow and Genie excelled in generating short proteins. RFdiffusion emerged as the most robust model across the widest range of protein lengths.
- Efficiency: There is a stark contrast in computational demand. FrameFlow can generate backbones in approximately 0.15 minutes, whereas models like Genie can take up to 8 minutes per structure.
- Structural Bias: A pervasive "helical bias" was observed across all models, with many structures consisting of over 50% alpha-helices. This suggests that current architectures struggle to capture the complex, long-range interactions required for beta-sheets and flexible loops.
Conditional Generation (Motif-Scaffolding)
In tasks requiring the design of a scaffold around a functional motif, GPDL-H (an ESM-based model) and RFdiffusion showed the strongest performance, achieving success rates over 50%. The primary bottleneck in this area is not the quality of the scaffold itself, but the failure of models to maintain the precise geometry of the functional motif during the generation process.
Discussion and Future Directions
Scaffold-Lab highlights several critical hurdles for the next generation of protein design models. First, scaling to large, multi-domain proteins (>800 residues) remains a significant challenge. Second, there is a dire need for models that can move beyond rigid helical bundles to generate more diverse topologies, including intrinsically disordered regions. Finally, as the field moves toward designing complex enzymes and dynamic proteins, benchmarks must evolve to assess functional activity and conformational flexibility. All data and scripts for Scaffold-Lab are available at https://github.com/Immortals-33/Scaffold-Lab.
Summary updated