TL;DR
This paper introduces LM-GVP, a novel deep learning framework that combines protein language models (LMs) with graph neural networks (GNNs) to predict protein properties by leveraging both amino acid sequences and 3D structures. LM-GVP outperforms existing methods on various prediction tasks and offers insights into how structural information can enhance protein LMs.
What Problem Does This Paper Address?
Predicting protein properties is crucial for understanding biological functions and developing new therapeutics. While protein sequences hold a lot of information, they don't fully determine a protein's behavior, as 3D structure plays a vital role. Existing methods often treat sequence and structure information separately or rely on simplified structural representations. This paper addresses the challenge of effectively integrating rich 3D structural information with powerful protein language models to achieve more accurate and comprehensive protein property predictions.
Methods
The core of this work is the LM-GVP framework, which integrates a protein Language Model (LM) with a Geometric Vector Perceptron (GVP) network, a type of Graph Neural Network (GNN).
- Protein LM: A pre-trained LM (like ProtBERT) is used to generate embeddings for individual amino acids (AAs) from the protein sequence.
- GNN (GVP): Protein 3D structures are represented as graphs where AAs are nodes and edges connect residues based on proximity. These graphs are enriched with both scalar (e.g., dihedral angles, AA identity) and vector (e.g., directional vectors) features. The GVP network is specifically designed to handle these geometric features and learn from the graph structure.
- End-to-End Training: Crucially, LM-GVP is trained end-to-end. This means gradients from the GNN's predictions are back-propagated not only through the GNN but also into the protein LM. This allows the LM to be fine-tuned to produce embeddings that are more sensitive to structural information, effectively injecting structural "inductive bias" into the LM.
- Comparison Models: The authors also compare LM-GVP against sequence-only baselines (fine-tuned LMs) and structure-only baselines (GNNs trained on structural graphs). They also evaluate "2-stage" models where the LM and GNN are trained independently.
Datasets
The study utilizes several publicly available datasets for training and evaluation:
- Gene Ontology (GO) Dataset: This dataset comprises three hierarchies of protein functions: Biological Processes (BP), Molecular Functions (MF), and Cellular Components (CC). It's used for multi-label classification tasks.
- TAPE (Tasks Assessing Protein Embeddings) Dataset: This includes two protein engineering tasks:
- Fluorescence: Predicting the log-fluorescence intensity of green fluorescent protein (GFP) mutants.
- Protease Stability: Measuring the stability of proteins against protease concentration. These are regression tasks.
Key Insights and Ideas
- Synergistic Value of Sequence and Structure: The research strongly supports the idea that combining sequence and 3D structure information leads to superior protein property predictions compared to using either alone.
- Structural Fine-Tuning of LMs: The end-to-end training of LM-GVP allows structural information to actively "fine-tune" the protein LM. This is a novel approach that goes beyond simply using LM embeddings as input to a GNN.
- GVP's Geometric Sensitivity: The GVP network, with its ability to handle vector features and rotational equivariance, proves more effective than standard Graph Attention Networks (GATs) for leveraging detailed structural information.
- Interpretability Reveals Functional Sites: Using Integrated Gradients, the authors show that LM-GVP can identify residues crucial for specific protein functions (like ATP binding), aligning well with known active sites. This suggests the model learns biologically relevant features.
- LM Structure Capacity is Limited: The observation that fine-tuning LMs solely on sequence data can sometimes reduce their ability to retain structural information (as seen in contact map prediction) suggests that LMs alone have a limited capacity to capture complex structural nuances without explicit structural guidance.
Key Results
- State-of-the-Art Performance: LM-GVP consistently outperformed sequence-only and structure-only baselines, as well as 2-stage models, across all GO functional prediction tasks (Table 1) and achieved competitive or superior results on Fluorescence and Protease stability tasks (Table 2).
- GO Term Prediction: LM-GVP showed the best performance on GO terms, with the majority of GO terms (82.7%) being better predicted by LM-GVP than baselines. Molecular Function (MF) terms were generally more predictable than Cellular Component (CC) or Biological Process (BP) terms.
- Interpretability Success: LM-GVP's saliency maps, derived from Integrated Gradients, showed good agreement with known binding sites for ATP, GTP, and heme binding proteins, demonstrating its ability to pinpoint functionally important residues.
- Preservation of Structural Information: LM-GVP fine-tuning better preserved the protein LM's intrinsic ability to predict contact maps compared to sequence-only fine-tuning, especially for GO tasks.
Limitations and Future Work
- Limited Structural Data: The authors acknowledge that the availability of 3D protein structures is significantly lower than that of sequences, posing a challenge for structure-informed models.
- Contact Map Prediction Nuances: While LM-GVP preserves contact map prediction ability better than sequence-only fine-tuning, it's not significantly better for protein engineering tasks. The paper notes that contact maps are only one aspect of 3D structure.
- Generalizability to Natural Language: The authors suggest that the principles of LM-GVP, particularly the end-to-end integration of GNNs and LMs, could be extended to natural language processing tasks.
- Further Exploration of Structural Features: While GVP handles rich structural features, future work could explore even more detailed geometric representations or different GNN architectures.
Summary updated