Highly accurate and robust protein sequence design with CarbonDesign

Haicang Zhang, Milong Ren, Chungong Yu, Dongbo BuResearch Square 2023 · DOI

Abstract

Protein sequence design, the inverse problem of protein structure prediction, plays a crucial role in protein engineering. Although recent deep learning-based methods have shown promising advancements, achieving accurate and robust protein sequence design remains an ongoing challenge. Here, we present CarbonDesign, a new approach that draws inspiration from successful ingredients of AlphaFold for protein structure prediction and makes significant and novel developments tailored specifically for protein sequence design. At its core, CarbonDesign explores Inverseformer, a novel network architecture adapted from AlphaFold's Evoformer, to learn representations from backbone structures and an amortized Markov Random Fields model for sequence decoding. Moreover, we incorporate other essential AlphaFold concepts into CarbonDesign: an end-to-end network recycling technique to leverage evolutionary constraints in protein language models and a multi-task learning technique to generate side chain structures corresponding to the designed sequences. Through rigorous evaluations of independent testing data sets, including the CAMEO and recent CASP15 data sets, as well as the predicted structures from AlphaFold, we show that CarbonDesign outperforms other published methods, achieving high accuracy in sequence generation. Moreover, it exhibits superior performance on de novo backbone structures obtained from recent diffusion generative models such as RFdiffusion and FrameDiff, highlighting its potential for enhancing de novo protein design. Notably, CarbonDesign also supports zero-shot prediction of the functional effects of sequence variants, indicating its potential application in directed evolution-based design. In summary, our results illustrate CarbonDesign's accurate and robust performance in protein sequence design, making it a promising tool for applications in bioengineering.

Introduction

Protein sequence design, often referred to as "inverse folding," is the challenge of identifying an amino acid sequence that will fold into a specific, pre-defined 3D backbone structure. This task is fundamental to protein engineering, enabling the creation of novel enzymes, therapeutics, and synthetic materials. While tools like ProteinMPNN and ESM-IF have advanced the field, they often struggle to integrate evolutionary information effectively or handle the structural "noise" inherent in synthetic backbones generated by diffusion models like RFdiffusion. CarbonDesign addresses these gaps by adapting the architectural principles of AlphaFold to the inverse problem.

Methods

CarbonDesign introduces several technical innovations to improve sequence design accuracy and robustness.

Overview of CarbonDesign architecture: (a) the full pipeline from 3D backbone to sequence, (b) details of the Inverseformer block, and (c) the multi-task learning heads.

The framework consists of several core components:

  • Inverseformer: Adapted from AlphaFold’s Evoformer, this module processes 3D structural features—such as local orientations and residue distances—through 12 blocks of node and triangular edge updates. It maps 3D geometry into high-dimensional representations suitable for sequence prediction.
  • Amortized Markov Random Fields (MRF): Instead of simple autoregressive decoding, CarbonDesign utilizes an MRF-Sequence Module. This treats sequence design as a probabilistic graphical model problem, capturing complex pairwise dependencies between amino acids.
  • End-to-End Recycling with ESM2: The model employs a "recycling" strategy where intermediate outputs are refined through multiple passes. Crucially, it incorporates embeddings from the ESM2 protein language model during these cycles, leveraging evolutionary "common sense" alongside structural constraints.
  • Multi-task Learning: The network simultaneously predicts the amino acid sequence and the 3D coordinates of side-chain atoms. This auxiliary task encourages the model to learn physically realistic representations of the protein's internal environment.

Results and Discussion

CarbonDesign was evaluated against standard benchmarks (CAMEO, CASP15), orphan proteins, and de novo structures from diffusion models.

Performance and Robustness

CarbonDesign consistently outperformed state-of-the-art (SOTA) methods. It achieved a 60.1% sequence recovery on the CAMEO dataset, compared to 54.8% for ESM-IF and 51.3% for ProteinMPNN. On de novo backbones from RFdiffusion, CarbonDesign showed a substantial improvement in self-consistency TM-score (scTM), indicating that its designed sequences are more likely to fold into the intended target shapes.

Functional Prediction

In zero-shot prediction of mutation effects, CarbonDesign achieved a Spearman correlation of 0.43, outperforming language models like ESM-1v. It also demonstrated high precision (auROC > 0.95) in distinguishing pathogenic from benign variants in critical genes like TP53.

Theoretical Grounding

A key insight of this work is the connection between the triangular edge updates in the Inverseformer and the Belief Propagation algorithm. This provides a theoretical basis for the architecture's effectiveness in resolving complex spatial constraints. Furthermore, the integration of ESM2 embeddings acts as a powerful prior, ensuring designed sequences remain biologically plausible even when target structures are slightly unnatural.

Conclusion

CarbonDesign represents a significant step forward in robust protein sequence design. By "inverting" the AlphaFold architecture and incorporating large-scale language model priors, it achieves superior accuracy across both natural and synthetic protein backbones. Future work will focus on extending the model to handle multi-chain complexes and validating designed proteins through experimental "wet-lab" synthesis.

Summary updated

Abstract

<title>Abstract</title> Protein sequence design, the inverse problem of protein structure prediction, plays a crucial role in protein engineering. Although recent deep learning-based methods have shown promising advancements, achieving accurate and robust protein sequence design remains an ongoing challenge. Here, we present CarbonDesign, a new approach that draws inspiration from successful ingredients of AlphaFold for protein structure prediction and makes significant and novel developments tailored specifically for protein sequence design. At its core, CarbonDesign explores Inverseformer, a novel network architecture adapted from AlphaFold's Evoformer, to learn representations from backbone structures and an amortized Markov Random Fields model for sequence decoding. Moreover, we incorporate other essential AlphaFold concepts into CarbonDesign: an end-to-end network recycling technique to leverage evolutionary constraints in protein language models and a multi-task learning technique to generate side chain structures corresponding to the designed sequences. Through rigorous evaluations of independent testing data sets, including the CAMEO and recent CASP15 data sets, as well as the predicted structures from AlphaFold, we show that CarbonDesign outperforms other published methods, achieving high accuracy in sequence generation. Moreover, it exhibits superior performance on <italic>de novo</italic> backbone structures obtained from recent diffusion generative models such as RFdiffusion and FrameDiff, highlighting its potential for enhancing <italic>de novo</italic> protein design. Notably, CarbonDesign also supports zero-shot prediction of the functional effects of sequence variants, indicating its potential application in directed evolution-based design. In summary, our results illustrate CarbonDesign's accurate and robust performance in protein sequence design, making it a promising tool for applications in bioengineering.
BibTeX
@inproceedings{zhang2023highly,
  author = {Haicang Zhang and Milong Ren and Chungong Yu and Dongbo Bu},
  title = {{Highly accurate and robust protein sequence design with CarbonDesign}},
  booktitle = {Research Square},
  year = {2023},
  doi = {10.21203/rs.3.rs-3251939/v1},
}
Figure