Protein engineering in the deep learning era

Bingxin Zhou, Yang Tan, Yutong Hu, Lirong Zheng, Bozitao Zhong, Liang HongmLife 2024 · DOI

TL;DR

This review provides a comprehensive overview of how deep learning is revolutionizing protein engineering. It details the latest advancements in representing protein sequences and structures, and how these representations are used for tasks like predicting mutations and designing novel proteins, ultimately accelerating biological discovery and applications.

What Problem Does This Paper Address?

Protein engineering, a field crucial for advancements in healthcare, industrial production, and sustainability, has historically relied on time-consuming and expensive experimental methods like rational design and mutagenesis. While these methods have yielded successes, they are limited by the sheer complexity of proteins and the vastness of their potential variations. The challenge lies in efficiently understanding and manipulating protein sequences and structures to achieve desired functions. This paper addresses the need for a systematic approach to leverage the power of deep learning to overcome these limitations, providing a guide for both biologists and computer scientists to accelerate protein engineering research and applications.

Methods

The paper outlines a three-step deep learning pipeline for protein engineering:

  1. Data Preparation & Numerical Characterization: Proteins are first converted into numerical representations. This can be as simple as a sequence of amino acid tokens or more complex representations like graphs, 3D grids, or meshes, depending on the task.
  2. Model Pre-training for Representation Learning: This is a crucial step, especially given the scarcity of labeled experimental data. The paper discusses two main categories of models:
  3. Protein Language Models (PLMs): These models, inspired by Natural Language Processing (NLP), treat amino acid sequences as "words" in a sentence. They use architectures like Transformers to learn relationships between amino acids. Common pre-training tasks include Masked Language Modeling (MLM), where the model predicts masked amino acids. Encoder-only models (like BERT variants) are used for understanding and representation, decoder-only models (like GPT variants) for generation, and encoder-decoder models for both.
  4. Geometric Deep Learning Models: These models are designed to handle protein structures. They employ techniques like Graph Neural Networks (GNNs) for graph-based representations and Convolutional Neural Networks (CNNs) for grid-based (voxel) or mesh representations. These models leverage inductive biases like symmetry to better capture spatial relationships.
  5. Downstream Task Prediction: The learned representations are then used for specific protein engineering tasks. This can involve zero-shot predictions (without fine-tuning) or fine-tuned predictions using labeled data.

The paper also highlights the importance of multimodal and multiscale joint encoding, integrating information from sequences, structures, and other biological data sources to achieve a more holistic understanding of protein function.

Datasets

The paper extensively discusses various benchmark datasets used for training and evaluating deep learning models in protein engineering. These are categorized by task:

  • General Benchmarks: PETA, PEER, ProteinGLUE, BioMap. These offer a broad range of tasks for evaluating general protein representation learning.
  • Mutation Effect Prediction: ProteinGym (v1), DeepSequence, Envision, FLIP, FS-mutant, SoluProtMutDB, DTM, DDG, FireProtDB, ProThermDB. These datasets are crucial for understanding how mutations affect protein properties.
  • Structural Homology Detection: PDB2, CATH-Plus, AlphaFold DB, Malidup, Malisam, SWISS-MODEL, Pfam, InterPro, ModBase. These are used for tasks related to identifying similar protein structures and families.
  • Protein Solubility Prediction: TargetTrack, PROSO II, DeepSol, DeepSoluE, DeepSoluE, Protein-Sol, SoluProt.
  • Protein Localization Prediction: DeepLoc (1 & 2), LOCATE, WOLF PSORT, iLoc-Euk, Hum-mPLoc, Plant-mPLoc.
  • Protein Annotation Prediction: Gene Ontology, Enzyme Commission, PANTHER, BioCyc, Swiss-Prot, UniProt.
  • Compound-Protein Interaction: BindingDB, ChEMBL, PDBbind-CN, KLIFS, DrugBank, BioLiP.
  • Protein-Protein Interaction: STRING, BioGRID, IntAct, HPRD.

The paper does not introduce new benchmarks but rather consolidates and categorizes existing ones, emphasizing their importance for standardized evaluation.

Key Insights and Ideas

  • The "Central Dogma" of Protein Engineering: The paper frames protein engineering around the fundamental relationship between sequence, structure, and function, mirroring the central dogma of molecular biology. This provides a clear conceptual framework for applying deep learning.
  • The Power of Self-Supervised Learning: Given the limited availability of labeled experimental data, self-supervised pre-training (especially using protein language models) is paramount. It allows models to learn rich, generalizable representations from vast amounts of unlabeled sequence data.
  • Bridging Sequence and Structure: The review highlights the complementary strengths of protein language models (for sequences) and geometric deep learning (for structures). Integrating these approaches is key to unlocking more comprehensive protein understanding and engineering capabilities.
  • Zero-Shot Prediction as a Game-Changer: The ability of pre-trained models to perform tasks with high accuracy without explicit task-specific training (zero-shot prediction) is a significant advancement, enabling rapid screening and hypothesis generation.
  • The Need for Interdisciplinary Collaboration: The paper strongly emphasizes that successful protein engineering with deep learning requires a close collaboration between biologists and computer scientists, combining domain expertise with algorithmic innovation.

Key Results

The paper is a review and does not present novel experimental results. However, it summarizes the state-of-the-art in deep learning for protein engineering, highlighting:

  • Improved Prediction Accuracy: Deep learning models, particularly those leveraging large-scale pre-training, have demonstrated significant improvements in various downstream tasks, including mutation effect prediction, protein function prediction, and structural analysis.
  • Accelerated Discovery: The ability to perform virtual screening, identify promising mutations, and design novel proteins computationally significantly reduces the time and cost associated with experimental validation.
  • Enhanced Understanding: Deep learning models are uncovering complex relationships between protein sequence, structure, and function that were previously difficult to discern, leading to a deeper understanding of biological mechanisms.
  • Broad Applicability: The reviewed methods are applicable across a wide range of protein engineering challenges, from developing new enzymes and therapeutics to understanding disease mechanisms.

Limitations and Future Work

  • Explainability of Deep Learning Models: A significant challenge is the "black box" nature of large deep learning models. Understanding why a model makes a certain prediction is crucial for building trust and ensuring reliability, especially in applications related to human health. Tools like SHAP and attention scores are mentioned as potential solutions.
  • Data Scarcity and Quality: While self-supervised learning mitigates the need for labeled data, the quality and quantity of available experimental data for specific tasks (e.g., protein properties under varying environmental conditions) remain a limitation.
  • Alignment Across Modalities: Integrating information from diverse data sources (genomic, transcriptomic, proteomic, experimental conditions) effectively is an ongoing challenge.
  • Ethical Considerations: The paper touches upon the dual-use potential of protein engineering technologies, such as the development of more virulent viruses, and emphasizes the need for careful evaluation and responsible data sharing.
  • Future Directions: The authors foresee deeper integration between biology and computer science communities, the development of more sophisticated multimodal and multiscale models, and the application of techniques like "unlearning" to mitigate the prediction of harmful properties. The ultimate goal is to unleash the full potential of deep learning for scientific breakthroughs in protein engineering.

Summary updated

Abstract

Advances in deep learning have significantly aided protein engineering in addressing challenges in industrial production, healthcare, and environmental sustainability. This review frames frequently researched problems in protein understanding and engineering from the perspective of deep learning. It provides a thorough discussion of representation methods for protein sequences and structures, along with general encoding pipelines that support both pre-training and supervised learning tasks. We summarize state-of-the-art protein language models, geometric deep learning techniques, and the combination of distinct approaches to learning from multi-modal biological data. Additionally, we outline common downstream tasks and relevant benchmark datasets for training and evaluating deep learning models, focusing on satisfying the particular needs of protein engineering applications, such as identifying mutation sites and predicting properties for candidates' virtual screening. This review offers biologists the latest tools for assisting their engineering projects while providing a clear and comprehensive guide for computer scientists to develop more powerful solutions by standardizing problem formulation and consolidating data resources. Future research can foresee a deeper integration of the communities of biology and computer science, unleashing the full potential of deep learning in protein engineering and driving new scientific breakthroughs.
BibTeX
@inproceedings{zhou2024protein,
  author = {Bingxin Zhou and Yang Tan and Yutong Hu and Lirong Zheng and Bozitao Zhong and Liang Hong},
  title = {{Protein engineering in the deep learning era}},
  booktitle = {mLife},
  year = {2024},
  doi = {10.1002/mlf2.12157},
}
Figure