RBP-TSTL is a two-stage transfer learning framework for genome-scale prediction of RNA-binding proteins

Xinxin Peng, Xiaoyu Wang, Yuming Guo, Zongyuan Ge, Fuyi Li, Xin Gao, Jiangning Song

TL;DR

This paper introduces RBP-TSTL, a novel two-stage transfer learning framework that significantly improves the prediction of RNA-binding proteins (RBPs). By leveraging knowledge from large, self-supervised pre-trained protein language models and then fine-tuning on specific RBP datasets, RBP-TSTL achieves state-of-the-art performance across multiple species, enabling more accurate genome-scale RBP identification.

What Problem Does This Paper Address?

RNA-binding proteins (RBPs) are crucial regulators of gene expression, playing vital roles in cellular processes and disease. Accurately identifying these proteins is essential for understanding biological mechanisms and for applications like disease diagnosis. Traditional experimental methods for identifying RBPs are time-consuming and costly, making them unsuitable for high-throughput screening. While computational methods have advanced, they often struggle with:

  • Species-specific characteristics: RBPs can exhibit unique features across different species, meaning models trained on one species may not perform well on another.
  • Data scarcity: Obtaining large, well-annotated datasets for RBP prediction, especially for less-studied species, remains a challenge.
  • Feature engineering limitations: Relying on hand-crafted features can be laborious and may not fully capture the complex sequence-structure-function relationships of RBPs.

This paper aims to overcome these limitations by developing a robust and accurate computational framework for predicting RBPs that is adaptable to different species and leverages the power of modern deep learning techniques.

Methods

The core of the RBP-TSTL framework is a two-stage transfer learning strategy:

  1. First Stage: Self-Supervised Pre-training Embeddings: The framework utilizes a pre-trained protein language model (specifically, ProtT5-XL) to generate rich, context-aware embeddings from protein sequences. This stage leverages the vast knowledge encoded in large protein sequence databases without requiring specific RBP annotations. These embeddings serve as powerful, general-purpose representations of protein sequences.
  2. Second Stage: Customized Deep Learning and Fine-tuning:
    • Initialization: A customized deep learning model, consisting of six Gated Recurrent Unit (GRU) layers and a discriminator, is initialized using the embeddings generated in the first stage.
  3. Annotated Pre-training: This model is first trained on a general, annotated RBP dataset (combining data from multiple species).
  4. Species-Specific Fine-tuning: Crucially, the model is then fine-tuned on each target species' specific RBP dataset. This fine-tuning step allows the model to adapt the general RBP knowledge to the unique characteristics of each species.

The model employs a weighted binary cross-entropy loss function to handle the inherent imbalance between positive (RBP) and negative (non-RBP) samples in the datasets.

Datasets

The study utilized several datasets:

  • Benchmark and Annotated Pre-training Datasets: These were sourced from a previous study [36] and included species-specific datasets for Homo sapiens, Arabidopsis thaliana, Escherichia coli, and Salmonella. A larger, annotated pre-training dataset encompassing all available species from Swiss-Prot was also used. Positive samples were identified using QuickGO API, and negative samples were carefully selected to exclude known nucleotide-binding proteins and those with annotated or potential RNA-binding domains (RBDs) from Pfam. Redundancy was removed using CD-HIT at a 25% similarity threshold.
  • Additional Validation Sets: Two sets of experimentally identified RBPs were used for validation, curated from previous studies [53, 14]. These sets were specifically designed to exclude proteins identified solely by the presence of RBDs.

Key Insights and Ideas

  • The Power of Pre-trained Language Models: The study strongly demonstrates that leveraging embeddings from large, self-supervised protein language models (like ProtT5-XL) provides a significant advantage over traditional hand-crafted features for RBP prediction. These embeddings capture complex sequence patterns that are difficult to engineer manually.
  • Two-Stage Transfer Learning is Key: The RBP-TSTL framework's success hinges on its two-stage approach. The first stage provides a robust, general representation, while the second stage effectively adapts this knowledge to species-specific nuances, mitigating the issue of data scarcity and species-specific characteristics.
  • Species-Specific Models are Still Important, but Transfer Learning Bridges Gaps: While species-specific models generally outperform cross-species models, RBP-TSTL shows that effective transfer learning can significantly improve cross-species prediction, suggesting that pre-trained embeddings retain valuable information across evolutionary distances.
  • Genome-Scale Prediction is Feasible and Valuable: The framework's ability to perform accurate genome-scale predictions opens doors for comprehensive RBP discovery and annotation across various organisms.

Key Results

  • State-of-the-Art Performance: RBP-TSTL consistently outperformed nine other state-of-the-art sequence-based RBP prediction methods across all four tested species (H. sapiens, A. thaliana, E. coli, Salmonella) on independent test datasets. For example, on the Salmonella test set, RBP-TSTL achieved an AUPRC of 0.90, significantly higher than other methods.
  • Improved Accuracy on Scarce Data: The framework showed superior performance even for species with limited annotated RBP data (e.g., E. coli and Salmonella), highlighting the effectiveness of transfer learning.
  • Validation on Additional Datasets: RBP-TSTL demonstrated high accuracy in retrieving experimentally validated RBPs from additional validation sets, achieving a TPR of 96.30% on one set.
  • Genome-Scale Predictions: The best-performing RBP-TSTL models were used to predict putative RBPs across the genomes of the four species, resulting in hundreds of thousands of predicted candidates. These results are publicly available as a computational compendium.

Limitations and Future Work

  • Cross-Species Transfer: While RBP-TSTL shows promise in cross-species prediction, the authors suggest further improvements could be made by specifically transferring knowledge within eukaryotic or prokaryotic species. For instance, fine-tuning a model trained on H. sapiens for A. thaliana (both eukaryotes) or E. coli for Salmonella (both prokaryotes) could be more effective than cross-domain transfer.
  • Data Imbalance Handling: The study used weighted binary cross-entropy for data imbalance. The authors suggest exploring alternative methods like bootstrapping for potentially better handling of severe data imbalance.
  • Model Architecture: The ablation study indicated that a six-layer GRU network provided a balanced performance. However, exploring different GRU layer configurations or entirely different network architectures could be a future direction.
  • Feature Encoding Comparison: The paper compared RBP-TSTL with models using hand-crafted features, confirming the superiority of LM-based embeddings. Future work could explore combining LM embeddings with other types of features for potentially even greater gains.

Summary updated

Abstract

RNA binding proteins (RBPs) are critical for the post-transcriptional control of RNAs and play vital roles in a myriad of biological processes, such as RNA localization and gene regulation. Therefore, computational methods that are capable of accurately identifying RBPs are highly desirable and have important implications for biomedical and biotechnological applications. Here, we propose a two-stage deep transfer learning-based framework, termed RBP-TSTL, for accurate prediction of RBPs. In the first stage, the knowledge from the self-supervised pre-trained model was extracted as feature embeddings and used to represent the protein sequences, while in the second stage, a customized deep learning model was initialized based on an annotated pre-training RBPs dataset before being fine-tuned on each corresponding target species dataset. This two-stage transfer learning framework can enable the RBP-TSTL model to be effectively trained to learn and improve the prediction performance. Extensive performance benchmarking of the RBP-TSTL models trained using the features generated by the self-supervised pre-trained model and other models trained using hand-crafting encoding features demonstrated the effectiveness of the proposed two-stage knowledge transfer strategy based on the self-supervised pre-trained models. Using the best-performing RBP-TSTL models, we further conducted genome-scale RBP predictions for Homo sapiens, Arabidopsis thaliana, Escherichia coli, and Salmonella and established a computational compendium containing all the predicted putative RBPs candidates. We anticipate that the proposed RBP-TSTL approach will be explored as a useful tool for the characterization of RNA-binding proteins and exploration of their sequence–structure–function relationships.
BibTeX
@article{peng2022rbp,
  author = {Xinxin Peng and Xiaoyu Wang and Yuming Guo and Zongyuan Ge and Fuyi Li and Xin Gao and Jiangning Song},
  title = {{RBP-TSTL is a two-stage transfer learning framework for genome-scale prediction of RNA-binding proteins}},
  journal = {Briefings in Bioinformatics},
  year = {2022},
  url = {https://openreview.net/forum?id=e5oqiBWhv7Z},
}
Figure