SAINT-Angle: self-attention augmented inception-inside-inception network and transfer learning improve protein backbone torsion angle prediction

A.k.m. Mehedi Hasan, Ajmain Yasar Ahmed, Sazan Mahbub, Mohammad Saifur Rahman, Md. Shamsuzzoha BayzidbioRxiv 2022 · DOI

TL;DR

This paper introduces SAINT-Angle, a novel deep learning model that significantly improves the prediction of protein backbone torsion angles, crucial for understanding protein structure and function. By augmenting a self-attention network with transfer learning from large language models, SAINT-Angle achieves state-of-the-art accuracy on benchmark datasets, outperforming existing methods.

What Problem Does This Paper Address?

Determining the 3D structure of proteins is fundamental to understanding their function and interactions. A key step in this process is predicting protein backbone torsion angles (phi and psi), which define the local conformation of the protein chain. Traditional experimental methods for structure determination are time-consuming and expensive. While computational methods have advanced, accurately predicting these angles, especially considering long-range interactions within the protein, remains a challenge. Existing methods often struggle to capture these complex dependencies, limiting the overall accuracy of protein structure prediction.

Methods

The core of SAINT-Angle is an enhanced version of the SAINT architecture, originally designed for protein secondary structure prediction. The key innovations include:

  • Self-Attention Module: This module, inspired by transformer architectures, is crucial for capturing long-range dependencies between amino acid residues in a protein sequence.
  • Inception-Inside-Inception (2A3I) and Residual (RES-2A3I) Modules: These modules are used to effectively process local and global features, with residual connections added to improve training stability and enable deeper networks.
  • Transfer Learning with ProtTrans: The model leverages pre-trained large language models (ProtTrans) to extract powerful feature representations from protein sequences. This transfer learning approach significantly boosts performance.
  • Ensemble Approach: SAINT-Angle is an ensemble of eight diverse models, combining different architectures and feature sets to achieve robust and accurate predictions.

Datasets

The study utilized several established datasets for training, validation, and testing:

  • Training and Validation: The SPOT-1D dataset was used for training and hyperparameter tuning.
  • Testing: The models were evaluated on benchmark datasets including TEST2016, TEST2018, CAMEO, and CASP (CASP12, CASP13, CASP-FM). These datasets represent a diverse range of protein structures and sizes.

Key Insights and Ideas

  • Long-Range Interactions are Key: The self-attention mechanism's ability to capture long-range interactions proved critical for improving torsion angle prediction, especially as the complexity of these interactions increased.
  • Transfer Learning Pays Off: Utilizing pre-trained protein language models (ProtTrans) provided a significant performance boost, demonstrating the power of leveraging large-scale pre-training for downstream tasks.
  • Architecture Matters, but Features are Crucial: While the SAINT-Angle architecture is robust and performs well across different feature sets, the inclusion of ProtTrans features and evolutionary signatures (from ESIDEN) led to substantial improvements.
  • Ensembling for Robustness: Combining multiple diverse models in an ensemble significantly outperformed individual base models, highlighting the benefit of diverse learning paths.

Key Results

SAINT-Angle demonstrated state-of-the-art performance across multiple benchmark datasets:

  • Significant Improvement: SAINT-Angle consistently outperformed existing methods like OPUS-TASS, SPOT-1D, NetSurfP-2.0, MUFOLD, and SPIDER3.
  • MAE Reduction: The model achieved notable reductions in Mean Absolute Error (MAE) for both phi (φ) and psi (ψ) angles. For psi angles, the improvement was particularly substantial, with MAE(ψ) being more than 2-6 degrees lower than other methods on benchmark datasets.
  • Statistical Significance: The improvements were statistically significant (p-value < 0.05) across various comparisons.
  • ProtTrans Impact: Models incorporating ProtTrans features showed significant improvements over those using only base features, underscoring the value of transfer learning.

Limitations and Future Work

  • Feature Robustness vs. Simplicity: While SAINT-Angle is feature-robust, future work could explore selecting a minimal set of simple features that still achieve high accuracy.
  • Multi-Task Learning: The authors suggest extending the model to a multi-task learning framework that simultaneously predicts secondary structures and torsion angles.
  • Predicting Other Attributes: The proposed model could be adapted to predict other protein attributes beyond torsion angles.
  • Comparison with ESIDEN: While SAINT-Angle-Single (using ESIDEN features) showed strong performance, the paper notes that ESIDEN itself uses novel evolutionary features that SAINT-Angle (without them) doesn't directly incorporate. Further investigation into combining these feature sets could be beneficial.

Summary updated

Abstract

Abstract Motivation Protein structure provides insight into how proteins interact with one another as well as their functions in living organisms. Protein backbone torsion angles ( ϕ and ψ ) prediction is a key sub-problem in predicting protein structures. However, reliable determination of backbone torsion angles using conventional experimental methods is slow and expensive. Therefore, considerable effort is being put into developing computational methods for predicting backbone angles. Results We present SAINT-Angle, a highly accurate method for predicting protein backbone torsion angles using a self-attention based deep learning network called SAINT, which was previously developed for the protein secondary structure prediction. We extended and improved the existing SAINT architecture as well as used transfer learning to predict backbone angles. We compared the performance of SAINT-Angle with the state-of-the-art methods through an extensive evaluation study on a collection of benchmark datasets, namely, TEST2016, TEST2018, CAMEO, and CASP. The experimental results suggest that our proposed self-attention based network, together with transfer learning, has achieved notable improvements over the best alternate methods. Availability and implementation SAINT-Angle is freely available as an open-source project at https://github.com/bayzidlab/SAINT-Angle . Contact [email protected] Supplementary information Supplementary material SM.pdf.
BibTeX
@misc{hasan2022saint,
  author = {A.K.M. Mehedi Hasan and Ajmain Yasar Ahmed and Sazan Mahbub and Mohammad Saifur Rahman and Md. Shamsuzzoha Bayzid},
  title = {{SAINT-Angle: self-attention augmented inception-inside-inception network and transfer learning improve protein backbone torsion angle prediction}},
  year = {2022},
  doi = {10.1101/2022.12.08.519543},
  howpublished = {bioRxiv},
}
Figure