TL;DR
This paper introduces a novel deep learning approach that jointly encodes both protein sequence and 3D structure to create richer, more informative protein embeddings. This "structure-aware" embedding significantly improves performance on tasks like predicting the impact of mutations, demonstrating a more data-efficient way to leverage protein information for design and understanding.
What Problem Does This Paper Address?
Proteins are the workhorses of biology, performing nearly all critical functions in living organisms. Understanding the intricate relationship between a protein's linear amino acid sequence and its complex 3D structure is fundamental to predicting its function and designing new proteins for specific purposes. While powerful deep learning models have emerged that learn from vast amounts of protein sequence data (akin to natural language processing), these models often struggle with tasks where structural information is crucial. This is partly because experimentally determined protein structures are far fewer than sequences. This paper aims to bridge this gap by developing a method that effectively integrates both sequence and structural information into a unified protein representation, making it more data-efficient and biologically insightful.
Methods
The core innovation lies in a semi-supervised training framework that leverages a SE(3)-equivariant Transformer. This architecture is designed to process both 1D sequence embeddings (from the pre-trained ESM-1b model) and 2D structural features. The key is a "joint training" objective where the model is tasked with simultaneously predicting masked portions of both the protein sequence and its 3D structure. This multi-task learning forces the model to learn a unified embedding space that captures complex relationships between amino acids in both sequential and spatial contexts. The model is trained on masked sequence and structure recovery, and then fine-tuned for downstream tasks like predicting mutation effects.
Datasets
- trRosetta2: Used for training and validation of the model's ability to recover masked sequence and structure.
- ProTherm dataset: Used for fine-tuning and evaluating the model's performance in predicting the effect of single mutations on thermal stability (∆∆G values). This dataset was split into training and testing sets with non-overlapping wild-types.
- Predicted structures from RoseTTAFold: The model was also evaluated using predicted protein structures, demonstrating its robustness to approximate structural data.
Key Insights and Ideas
- Structure is a powerful prior: Explicitly incorporating structural information into protein embeddings acts as a strong biological prior, leading to more informative representations than sequence alone.
- Data efficiency: Jointly learning from sequence and structure allows for better performance with potentially less data compared to sequence-only models, addressing the scarcity of structural data.
- Capturing complex residue interactions: The structure-aware embeddings can identify not only sequentially close but also spatially close residues that are affected by a mutation, even if they are distant in the sequence. This highlights the model's ability to grasp intricate 3D relationships.
- Robustness to predicted structures: The model's ability to maintain high accuracy even when using predicted protein structures (from RoseTTAFold) is a significant finding, paving the way for broader applications in protein design where experimental structures are unavailable.
Key Results
- Masked Structure and Sequence Recovery: The model achieved accurate recovery of masked structure (reducing RMSD from ~5 Å to ~2 Å) and improved masked sequence recovery (13% accuracy compared to 6% for the sequence-only baseline).
- Predicting Single Mutation Effects: The joint embedding model showed a higher Spearman rank-order correlation coefficient for predicting ∆∆G values compared to the sequence-only baseline, indicating better prediction of mutation impacts on thermal stability.
- Structure-Awareness Demonstrated: Analysis of embedding space differences between wild-type and mutant proteins for PDB 1FXA revealed that the structure-aware model identified more relevant affected residues, aligning with known structural proximity.
- Performance with Predicted Structures: The model achieved a Spearman correlation of 0.64 when using predicted structures, comparable to its performance with experimentally validated structures, underscoring its generalizability.
Limitations and Future Work
- Model Size and Embedding Space: The authors suggest that increasing the model size and the dimensionality of the embedding space could allow it to store even more information.
- Atom-wise Corrections: For masked structure recovery, predicting corrections for each atom independently could be improved by treating atom orientation and displacement as interdependent.
- Exploring More Prediction Tasks: The authors are keen to explore the application of these joint embeddings to a wider range of prediction tasks, including predicting the deleteriousness of mutations using Deep Mutational Scanning (DMS) data.
- Iterative Refinement: Incremental corrections to structure coordinates over multiple iterations could further enhance masked recovery.
Summary updated