A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods

Haifeng Chen, Zhuoqi Zheng, Bo Zhang, Bozitao Zhong, Kexin Liu, Zhengxin Li, Jun‐Jie Zhu, Jia-Le Yu, Ting WeiResearch Square 2024 · DOI

Introduction

De novo protein design has transitioned from simple sequence optimization to the generation of entirely novel protein backbones. While generative AI models like RFdiffusion, FrameFlow, and Chroma have demonstrated remarkable potential, the field has lacked a standardized evaluation framework. Disparate datasets, varying protein lengths, and inconsistent metrics have made "apples-to-apples" comparisons nearly impossible. To address this, we introduce Scaffold-Lab, a unified benchmarking framework designed to rigorously assess protein backbone generation methods across designability, novelty, diversity, and efficiency.

The Scaffold-Lab Framework

Scaffold-Lab evaluates models through a comprehensive "refolding pipeline" that tests whether a generated backbone is physically viable. The process follows four critical steps:

  1. Backbone Generation: The generative model produces a 3D protein backbone.
  2. Sequence Design: ProteinMPNN is utilized to design an amino acid sequence compatible with the generated backbone.
  3. Structure Prediction: ESMFold predicts the 3D structure of the newly designed sequence.
  4. Validation: The predicted structure is compared against the original AI-generated backbone using the self-consistency TM-score (sc-TM). A design is considered successful if the sc-TM exceeds 0.5.
The overall workflow of Scaffold-Lab, illustrating the pipeline from backbone generation to refolding and evaluation.

Beyond basic designability, Scaffold-Lab introduces a Novelty Score with a Penalty to ensure models are not rewarded for generating un-foldable "hallucinations." The framework also employs TOPSIS, a multi-criteria decision-making method, to provide an objective ranking of model performance.

Benchmarking Results

Unconditional Generation

Our evaluation of seven state-of-the-art methods revealed a significant "designability watershed." While models perform exceptionally well for short proteins, success rates drop precipitously once proteins exceed 300 residues, with rates falling below 30% for proteins longer than 800 residues.

  • Top Performers: FrameFlow and Genie excelled in generating short proteins. RFdiffusion emerged as the most robust model across the widest range of protein lengths.
  • Efficiency: There is a stark contrast in computational demand. FrameFlow can generate backbones in approximately 0.15 minutes, whereas models like Genie can take up to 8 minutes per structure.
  • Structural Bias: A pervasive "helical bias" was observed across all models, with many structures consisting of over 50% alpha-helices. This suggests that current architectures struggle to capture the complex, long-range interactions required for beta-sheets and flexible loops.

Conditional Generation (Motif-Scaffolding)

In tasks requiring the design of a scaffold around a functional motif, GPDL-H (an ESM-based model) and RFdiffusion showed the strongest performance, achieving success rates over 50%. The primary bottleneck in this area is not the quality of the scaffold itself, but the failure of models to maintain the precise geometry of the functional motif during the generation process.

Discussion and Future Directions

Scaffold-Lab highlights several critical hurdles for the next generation of protein design models. First, scaling to large, multi-domain proteins (>800 residues) remains a significant challenge. Second, there is a dire need for models that can move beyond rigid helical bundles to generate more diverse topologies, including intrinsically disordered regions. Finally, as the field moves toward designing complex enzymes and dynamic proteins, benchmarks must evolve to assess functional activity and conformational flexibility. All data and scripts for Scaffold-Lab are available at https://github.com/Immortals-33/Scaffold-Lab.

Summary updated

Abstract

<title>Abstract</title> <italic>De novo</italic> protein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal that <italic>FrameFlow</italic> and <italic>RFdiffusion</italic> in unconditional generation and <italic>GPDL-H</italic> in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts are available at https://github.com/Immortals-33/Scaffold-Lab.
BibTeX
@inproceedings{chen2024unified,
  author = {Haifeng Chen and Zhuoqi Zheng and Bo Zhang and Bozitao Zhong and Kexin Liu and Zhengxin Li and Jun‐Jie Zhu and Jia-Le Yu and Ting Wei},
  title = {{A Unified Framework of Scaffold-Lab for Critical Assessment of Protein Backbone Generation Methods}},
  booktitle = {Research Square},
  year = {2024},
  doi = {10.21203/rs.3.rs-4249839/v1},
}
Figure