Homology detection using a protein secondary structure-based large language model

Roman Kogay, Weicheng Ma, Jad Bousselham, Zechen Yang, Daniel N. Rockmore, Olga Zhaxybayeva, Soroush VosoughibioRxiv 2023 · DOI

TL;DR

This paper introduces ProSSL, a novel deep learning model that leverages protein secondary structure information to detect distant protein homologies. By encoding secondary structure patterns, ProSSL achieves high accuracy in identifying related proteins, even those missed by traditional sequence-based methods, and offers interpretable insights into its predictions.

What Problem Does This Paper Address?

Understanding protein function is crucial for biology, and a primary method for doing so is identifying homologous proteins—those with shared ancestry and likely similar functions. Traditional methods rely on comparing amino acid sequences, but these can fail when proteins have diverged significantly over evolutionary time, accumulating many mutations. While 3D protein structures are more conserved and could help identify distant homologs, obtaining and processing this data is computationally expensive and limited in availability. This paper addresses the challenge of accurately detecting distant protein homologies by proposing a method that bridges the gap between sequence-based and structure-based approaches, using the more readily available and conserved secondary structure information.

Methods

The core of this work is the Protein Secondary Structure Language (ProSSL) model, built upon the Transformer architecture, a standard in large language models. Instead of using amino acid sequences directly, ProSSL takes tokenized representations of protein secondary structures (e.g., helix, strand, coil) as input. The model is trained using a pre-train/fine-tune schema.

  1. Pre-training: The model is first pre-trained on a large dataset of bacterial protein secondary structures using a masked language modeling (MLM) objective. This is analogous to how language models learn grammar and context from text. The model learns to predict masked secondary structure tokens based on their surrounding context.
  2. Fine-tuning: After pre-training, the model is fine-tuned on specific protein families (large terminase and portal proteins in this study) for classification tasks.
  3. Interpretation: The authors utilize Shapley Additive exPlanations (SHAP) values to interpret the model's predictions, allowing them to identify which regions of the secondary structure are most influential in determining homology.

Datasets

  • Pre-training: 334,881 bacterial protein sequences from the Swiss-Prot database were used. After filtering, 200,000 sequences were used for training and 134,881 for validation.
  • Fine-tuning:
    • Terminase prediction: 11,002 large terminase (TerL) sequences and 49,836 non-terminase sequences.
  • Portal protein prediction: 14,697 portal protein sequences and 49,827 non-portal protein sequences (derived from the non-terminase dataset).
  • Evaluation: The models were evaluated on curated datasets with various phylogenetic splits to ensure robustness.

Key Insights and Ideas

  • Secondary structure as a "language": The paper posits that protein secondary structure, like amino acid sequences, can be viewed as a "language" with its own syntax (arrangement of structural elements) that carries functional and evolutionary information.
  • Bridging sequence and structure: ProSSL effectively leverages secondary structure, which is more conserved than primary sequence but more readily available and predictable than tertiary structure, offering a powerful intermediate for homology detection.
  • Detecting "hidden" homologies: The model demonstrates a significant ability to identify distant homologs that are missed by traditional sequence-based methods like BLASTP, highlighting its utility for uncovering evolutionary relationships.
  • Interpretable predictions: The use of SHAP values allows researchers to understand why ProSSL makes a particular classification, pinpointing key secondary structure regions responsible for the decision, which can lead to functional insights.

Key Results

  • High Accuracy: Fine-tuned ProSSL models achieved average AUC scores of 98.71-99.96% for terminase prediction and 98.62-99.98% for portal protein prediction, comparable to or exceeding state-of-the-art primary sequence-based models like PROST.
  • Efficiency: ProSSL achieved comparable results using significantly less data (three orders of magnitude less) than some baseline models, making it "lightweight" and easier to fine-tune.
  • Identification of Novel Homologs: ProSSL successfully identified members of viral protein families that were missed by traditional amino acid similarity searches.
  • Structural Insights: SHAP value analysis revealed that ProSSL utilizes information across the entire protein for true positive matches, and for false positives, it can highlight structurally aligned regions that might indicate previously unrecognized homologies. For instance, it identified a potential portal protein based on structural similarity despite lacking sequence similarity.

Limitations and Future Work

  • Model Specificity: SHAP values are model-specific, and the authors suggest that future work should focus on developing model-agnostic interpretation metrics for broader comparison.
  • Domain-based Analysis: The current model is trained on full-length proteins. The authors suggest that training and fine-tuning on protein domains would expand its applicability to complex eukaryotic proteins with multiple domains.
  • Broader Applications: While demonstrated for homology detection, the authors believe ProSSL's ability to learn from secondary structure could benefit other tasks like signal peptide prediction.
  • Data Availability: While secondary structure is more accessible than tertiary structure, the availability of high-quality secondary structure predictions for all proteins remains a factor.

Summary updated

Abstract

Abstract Detection of homology among proteins is fundamental to understanding protein function. Unfortunately, traditional homology searches using amino acid sequence similarity are limited when numerous amino acid substitutions have accumulated either due to billions of years of evolution or through processes of accelerated change. Recent applications of deep-learning approaches demonstrate that “protein language” models of amino acid sequences can improve the accuracy of the traditional homology searches. Ultimately, the ability to work seamlessly with tertiary structures of proteins will solve the homology detection challenge and provide accompanying insights directly related to function, but to date the use of 3D structures suffers both from data availability and computational bottlenecks. Herein, we present the Protein Secondary Structure Language (ProSSL) model, an efficient encoding of protein secondary structure information in a Transformer-based deep-learning architecture. We conjecture that the secondary protein structure, which is better conserved than primary sequences and much more easily predictable and available than tertiary protein structure, could aid in the task of homology detection. ProSSL has the computational advantages of primary sequence-based homology detection, while also providing important structural information for similarity scoring. Using two case studies of large, diverse viral protein families, we show that the ProSSL model successfully captures patterns of secondary structure arrangements and is effective in detecting homologs either as a pre-trained or fine-tuned model. In both tasks, we accurately detect members of these protein families, including those missed in traditional amino acid similarity searches. We also illustrate how functional insights from the individual ProSSL models could be obtained from the use of the Shapley Additive exPlanations (SHAP) values. Author Summary When DNA is obtained from an organism or an environment, scientists are tasked with determining the functions of the proteins encoded in the genetic material. Such “functional annotation” relies on assigning functions based on the similarity of the proteins to counterparts in databases comprising annotated sequence data. Especially challenging is an ability to recognize similarity in proteins that accumulated a lot of amino acid changes. It is well-known that spatial structure of proteins that share ancestry and perform similar functions evolves much slower than the sequences of the proteins’ amino acids. Thus, comparison of 3D structures could address this challenge, but the data is still limited to certain classes of proteins, and the requisite computations are expensive. Herein we present a deep-learning model derived from protein secondary structure representation, a symbolic encoding of the way neighboring amino acid residues of a protein interact with each other. Unlike 3D structure, the secondary structure is quickly and accurately predictable from amino acid sequences of proteins. Using two viral proteins as case studies, we demonstrate that our model works well for detection of protein similarity, including identification of very distantly related proteins.
BibTeX
@misc{kogay2023homology,
  author = {Roman Kogay and Weicheng Ma and Jad Bousselham and Zechen Yang and Daniel N. Rockmore and Olga Zhaxybayeva and Soroush Vosoughi},
  title = {{Homology detection using a protein secondary structure-based large language model}},
  year = {2023},
  doi = {10.1101/2023.12.19.572443},
  howpublished = {bioRxiv},
}
Figure