Attention based convolutional neural network for predicting RNA-protein binding sites

Xiaoyong Pan, Junchi YanarXiv 2017 · OpenReview

TL;DR

This paper introduces iDeepA, an attention-based convolutional neural network (CNN) designed to accurately predict RNA-protein binding sites directly from RNA sequences. By incorporating an attention mechanism, iDeepA can better focus on crucial binding motifs within RNA, leading to comparable or improved performance over existing methods, especially for datasets with limited training data.

What Problem Does This Paper Address?

RNA-binding proteins (RBPs) are fundamental to numerous biological processes, including gene regulation. Identifying where these proteins bind to RNA molecules is critical for understanding their functions. While experimental methods like CLIP-seq exist, they are costly and time-consuming. Therefore, there's a pressing need for accurate and efficient computational methods to predict these binding sites. RBPs typically recognize specific sequence motifs on RNA. The challenge lies in computationally identifying these motifs and predicting binding sites, especially when dealing with variable RNA sequence lengths and the need to focus on the most informative parts of the sequence.

Methods

The core of iDeepA is a deep learning architecture that combines a Convolutional Neural Network (CNN) with an attention mechanism.

  1. One-Hot Encoding: RNA sequences are first converted into a one-hot encoded matrix, representing the presence of nucleotides A, C, G, and U.
  2. CNN: This matrix is fed into a CNN, which uses convolution, activation, and max-pooling operations to extract spatial features and generate feature maps.
  3. Attention Mechanism: Inspired by advancements in natural language processing, the paper introduces a two-level attention mechanism.
  4. Attention Model 1: Operates on the sequence dimension of the CNN's hidden states, allowing the model to focus on important positions along the RNA sequence.
  5. Attention Model 2: Operates on the feature map dimension (after transposing the hidden states), enabling the model to attend to different feature representations. Both attention models use a feedforward neural network to learn weights that highlight important parts of the input.

  6. Integration and Classification: The outputs from the CNN and the two attention models are concatenated and passed through two fully connected layers, culminating in a sigmoid layer for binary classification (bound vs. unbound site). The model is optimized using a categorical cross-entropy loss function with the RMSProp optimizer.

Datasets

The study utilizes a dataset derived from CLIP-seq experiments, sourced from GraphProt. This dataset comprises 24 experiments from 21 different RBPs. For each RBP, it includes thousands of bound RNA subsequences of varying lengths, along with a comparable number of negative sequences that are not known to be bound by that RBP. The authors mention that for evaluation, they ran their method and compared it with others on the same training and testing sets.

Key Insights and Ideas

  • Attention for Biological Relevance: The core idea is that not all parts of an RNA sequence are equally important for RBP binding. An attention mechanism can learn to "pay more attention" to the critical binding motifs, mimicking how biological systems might prioritize these regions.
  • Complementary Feature Extraction: Combining CNNs (for local feature extraction) with attention (for focusing on salient regions) provides a more comprehensive representation of RNA sequences for binding site prediction.
  • Handling Data Scarcity: The attention mechanism appears particularly beneficial for RBPs with smaller training datasets, suggesting it can improve generalization and learning from limited examples.

Key Results

iDeepA achieved an average Area Under the Curve (AUC) of 0.921 across 24 experiments, placing it among the top-performing methods. It demonstrated comparable performance to DeepBind (also 0.921 AUC) and outperformed GraphProt (0.887 AUC), deepnet-rbp (0.902 AUC), and MILCNN (0.861 AUC). Notably, iDeepA showed significant improvement for RBPs with small training sets. For instance, it achieved an AUC of 0.839 for C17ORF85 with only 4000 training samples, an 11% increase compared to DeepBind's 0.755 AUC on the same data. The authors suggest that for RBPs with large training sets, the additional attention mechanism might not offer substantial gains, as models can already converge to optimal solutions with sufficient data.

Limitations and Future Work

The authors acknowledge that while iDeepA performs well, they have not yet fully investigated whether the attention mechanism can be used to identify interpretable motifs. Future work aims to explore this interpretability further and to evaluate iDeepA more comprehensively on larger datasets and a wider range of RBPs.

Summary updated

Abstract

RNA-binding proteins (RBPs) play crucial roles in many biological processes, e.g. gene regulation. Computational identification of RBP binding sites on RNAs are urgently needed. In particular, RBPs bind to RNAs by recognizing sequence motifs. Thus, fast locating those motifs on RNA sequences is crucial and time-efficient for determining whether the RNAs interact with the RBPs or not. In this study, we present an attention based convolutional neural network, iDeepA, to predict RNA-protein binding sites from raw RNA sequences. We first encode RNA sequences into one-hot encoding. Next, we design a deep learning model with a convolutional neural network (CNN) and an attention mechanism, which automatically search for important positions, e.g. binding motifs, to learn discriminant high-level features for predicting RBP binding sites. We evaluate iDeepA on publicly gold-standard RBP binding sites derived from CLIP-seq data. The results demonstrate iDeepA achieves comparable performance with other state-of-the-art methods.
BibTeX
@misc{pan2017attention,
  author = {Xiaoyong Pan and Junchi Yan},
  title = {{Attention based convolutional neural network for predicting RNA-protein binding sites}},
  year = {2017},
  eprint = {1712.02270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1712.02270},
}
Figure