LM-GVP: A Generalizable Deep Learning Framework for Protein Property Prediction from Sequence and Structure

Zichen Wang, Steven A. Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel Oluwatobi Salawu, Colby J. Wise, Sri Priya Ponnapalli, Peter ClarkbioRxiv 2021 · DOI

TL;DR

This paper introduces LM-GVP, a novel deep learning framework that synergistically combines protein sequence information (via a language model) and 3D structural information (via a graph neural network) to predict protein properties. LM-GVP achieves state-of-the-art performance on various tasks, demonstrating the significant added value of integrating structural data for more accurate and generalizable protein property prediction.

What Problem Does This Paper Address?

Predicting protein properties (like function, stability, or interactions) is crucial for understanding biological systems and developing new therapeutics. While protein sequences contain a lot of information, they don't fully determine a protein's behavior. The 3D structure is key, but experimentally determining it is costly and time-consuming. Existing methods often treat sequence and structure information separately or rely on simplified structural representations. This paper addresses the challenge of effectively integrating rich, fine-grained 3D structural information with powerful protein language models (LMs) to unlock synergistic predictive capabilities for a wide range of protein property prediction tasks.

Methods

The core of this work is the LM-GVP framework, which elegantly connects a protein Language Model (LM) with a Geometric Vector Perceptron (GVP) network, a type of Graph Neural Network (GNN).

  • Protein Language Model (LM): The paper utilizes a pre-trained protein LM (specifically, ProtBERT) to process the 1D amino acid sequence and generate embeddings for each amino acid. These embeddings capture evolutionary and contextual information from the sequence.
  • Graph Neural Network (GNN - GVP): The 3D protein structure is first converted into a graph representation where amino acids are nodes and edges represent proximity. This graph is enriched with both scalar (e.g., dihedral angles, AA identity) and vector (e.g., directional information) features for nodes and edges. The GVP network is then employed to process this graph, learning complex structure-function relationships.
  • End-to-End Integration: The key innovation is the end-to-end training of the LM and GVP. Gradients from the GVP's predictions are back-propagated into the LM's transformer layers. This allows the LM to be fine-tuned not just on sequence but also with structural inductive biases, leading to more optimized sequence embeddings for property prediction. This is framed as a novel fine-tuning procedure for protein LMs.

The paper also compares LM-GVP against:

  • Sequence-only baselines: Fine-tuned protein LMs without structural information.
  • Structure-only baselines: GNNs (GAT and GVP) trained solely on structural graphs.
  • 2-stage models: Architectures where LM and GNN are trained independently.

Datasets

The study evaluated LM-GVP on three main types of datasets:

  • Gene Ontology (GO) dataset: This dataset comprises three hierarchies of protein functions: Biological Processes (BP), Molecular Functions (MF), and Cellular Components (CC). It was sourced from the DeepFRI project.
  • Protein Engineering Datasets (from TAPE):
    • Fluorescence: Predicting log-fluorescence intensity of green fluorescent protein (GFP) mutants.
  • Protease Stability: Measuring intrinsic protein stability against protease concentration.
  • Mutational Scans: Four datasets were used for analyzing the impact of mutations on protein function.

Key Insights and Ideas

  • Synergistic Power of Sequence and Structure: The most significant insight is that combining protein sequence and 3D structure information leads to superior performance compared to using either alone. This highlights that while LMs capture valuable evolutionary information, they have limitations in fully representing structural nuances, which are critical for function.
  • Structural Inductive Bias for LMs: LM-GVP's end-to-end training effectively injects structural "inductive bias" into the protein LM. This fine-tuning process guides the LM to generate sequence embeddings that are more attuned to structural properties, enhancing its predictive power.
  • GVP Network's Advantage: The Geometric Vector Perceptron (GVP) network, with its ability to handle rich scalar and vector features, proved to be more effective than simpler graph attention networks (GAT) for both structure-only and 2-stage models, underscoring the importance of detailed structural representations.
  • Interpretability Reveals Functional Sites: Using Integrated Gradients, the authors demonstrated that LM-GVP can identify residues crucial for specific molecular functions (like ATP, GTP, and heme binding) with higher accuracy than sequence-only or structure-only baselines, providing valuable mechanistic insights.
  • Preserving Structural Knowledge: LM-GVP fine-tuning helps protein LMs retain their intrinsic knowledge of protein contact maps, which is often lost when LMs are solely fine-tuned on sequence for specific downstream tasks.

Key Results

  • State-of-the-Art Performance: LM-GVP consistently outperformed sequence-only, structure-only, and 2-stage baseline models across all tested GO tasks (Table 1) and protein engineering tasks (Table 2).
  • GO Prediction: LM-GVP achieved the highest performance for GO-MF terms (micro-AUPR = 0.580), followed by GO-CC (0.423) and GO-BP (0.302). The vast majority of GO terms (82.7%) were predicted better by LM-GVP.
  • Protein Engineering: LM-GVP achieved competitive results on Fluorescence (Spearman's rho = 0.679) and Protease Stability (0.733), matching or exceeding the performance of the best baselines.
  • Interpretability: LM-GVP showed improved accuracy in identifying active sites for ATP, GTP, and heme binding compared to baselines, as measured by AUROC (Table 3).
  • Mutational Effects: LM-GVP fine-tuning generally preserved or improved the correlation of protein LM predictions with experimental mutational data across various assays, outperforming direct fine-tuning of LMs on specific tasks (Table 4).

Limitations and Future Work

  • Limited Protein Structures: The authors acknowledge the challenge of relatively fewer available protein structures (~180K) compared to sequences (>300B), which can limit the training of structure-aware models.
  • Residue Contact Map is One Aspect: While LM-GVP excels at preserving contact map information, the authors note that this is only one aspect of 3D structure, suggesting further exploration of other structural features could be beneficial.
  • Generalizability: The paper envisions LM-GVP being generalizable to many protein property prediction problems, implying that further testing on a broader range of tasks is a natural next step.
  • Extrapolation to Natural Language: The authors suggest that the approach of combining LMs with GNNs for fine-tuning could be extrapolated to natural language processing tasks, where graphs can represent linguistic structures.

Summary updated

Abstract

Abstract Proteins perform many essential functions in biological systems and can be successfully developed as bio-therapeutics. It is invaluable to be able to predict their properties based on a proposed sequence and structure. In this study, we developed a novel generalizable deep learning framework, LM-GVP, composed of a protein Language Model (LM) and Graph Neural Network (GNN) to leverage information from both 1D amino acid sequences and 3D structures of proteins. Our approach outperformed the state-of-the-art protein LMs on a variety of property prediction tasks including fluorescence, protease stability, and protein functions from Gene Ontology (GO). We also illustrated insights into how a GNN prediction head can guide the protein LM to better leverage structural information. We envision that our deep learning framework will be generalizable to many protein property prediction problems to greatly accelerate protein engineering and drug development.
BibTeX
@misc{wang2021lm,
  author = {Zichen Wang and Steven A. Combs and Ryan Brand and Miguel Romero Calvo and Panpan Xu and George Price and Nataliya Golovach and Emmanuel Oluwatobi Salawu and Colby J. Wise and Sri Priya Ponnapalli and Peter Clark},
  title = {{LM-GVP: A Generalizable Deep Learning Framework for Protein Property Prediction from Sequence and Structure}},
  year = {2021},
  doi = {10.1101/2021.09.21.460852},
  howpublished = {bioRxiv},
}
Figure