Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction

Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, Yuedong Yang

TL;DR

This paper introduces SpliceBERT, a novel language model for RNA sequences trained using self-supervised learning on millions of RNA sequences from 72 vertebrates. By leveraging evolutionary information across species, SpliceBERT significantly improves the prediction of RNA splicing events, including splice sites and branchpoints, and offers a powerful new tool for understanding the impact of genetic variants on splicing.

What Problem Does This Paper Address?

RNA splicing is a crucial process in gene expression, but predicting its intricacies from DNA sequences has been challenging. Existing computational models often struggle because they are trained on limited, single-species data, failing to capture the evolutionary conservation that plays a vital role in splicing. This limitation hinders our ability to accurately predict how genetic variations affect splicing, a key factor in many diseases. The authors aim to bridge this gap by developing a language model that can learn from a broad evolutionary spectrum of RNA sequences.

Methods

The core of this work is SpliceBERT, a language model built upon the BERT architecture. It's pre-trained using Masked Language Modeling (MLM), a self-supervised technique where the model learns to predict masked nucleotides in RNA sequences. Crucially, SpliceBERT is pre-trained on a massive dataset of primary RNA sequences from 72 vertebrates, a significant expansion beyond single-species models. This multi-species pre-training allows SpliceBERT to capture evolutionary conserved elements. The model's learned hidden states and attention weights are then analyzed to understand their biological relevance to splice sites. For downstream tasks, SpliceBERT is either used directly (zero-shot) or fine-tuned on specific prediction tasks.

Datasets

  • Pre-training Dataset: Over 2 million primary RNA sequences from 72 vertebrates, covering approximately 65 billion nucleotides.
  • Downstream Tasks Datasets:
    • Variant Effect Prediction: MFASS (27,733 variants) and Vex-seq (1,971 variants) datasets, focusing on splice-disrupting variants (SDVs).
  • Human Branchpoint (BP) Prediction: Mercer's dataset (55,739 high-confidence BPs and 921,518 non-BPs).
  • Cross-species Splice Site Prediction: The Spliceator dataset, covering over 100 eukaryotic species, with independent test sets for human, zebrafish, fruit fly, worm, and Arabidopsis.

Key Insights and Ideas

  • Multi-species pre-training is key: Training on a diverse set of vertebrate RNA sequences significantly enhances the model's ability to capture evolutionary conservation, which is critical for understanding biological functions like splicing.
  • Learned representations hold biological meaning: The hidden states and attention weights generated by SpliceBERT are not just abstract features; they reflect the biological properties of splice sites and can differentiate between functional and non-functional sites.
  • SSL for genomic sequences is powerful: Self-supervised learning, previously successful in protein sequences, proves highly effective for genomic sequences, enabling models to learn complex patterns without explicit labels.
  • Context matters for variant prediction: SpliceBERT's ability to capture contextual information around genetic variants leads to improved zero-shot prediction of their impact on RNA splicing.

Key Results

  • Evolutionary Conservation: SpliceBERT embeddings achieved a 0.353 AP score in distinguishing conserved from non-conserved sites, outperforming single-species models (SpliceBERT-human: 0.255).
  • Splice Site Characterization: SpliceBERT achieved the highest Normalized Mutual Information (NMI) score (0.31/0.31 for GT/AG) in distinguishing canonical splice sites from non-splice sites, significantly better than SpliceBERT-human (0.18/0.08).
  • Variant Effect Prediction: SpliceBERT-derived KL-context scores consistently outperformed baseline methods (including conservation scores and distance metrics) in classifying splice-disrupting variants in zero-shot settings.
  • Human Branchpoint Prediction: Fine-tuned SpliceBERT achieved an AP score of 0.745, outperforming SpliceBERT-human (0.730) and other specialized BP predictors.
  • Cross-species Splice Site Prediction: SpliceBERT showed superior performance (average 1.6-2.1% improvement in F1 score) compared to SpliceBERT-human and DNABERT on non-human test datasets.

Limitations and Future Work

  • MLM Accuracy: The balanced accuracy for MLM prediction is around 45-65%, suggesting room for improvement, potentially by increasing model size and pre-training data.
  • Limited Sequence Length: SpliceBERT currently handles sequences up to 1024 nt, while longer sequences are known to improve performance in some genomics tasks. Future work could explore more efficient attention mechanisms.
  • Tissue/Cell-Agnostic Pre-training: The current model does not account for tissue-specific splicing, which could be addressed by integrating experimental data.
  • Expanding Species Diversity: The authors suggest incorporating data from a wider range of organisms, including primates and mammals, to further enhance model capabilities.
  • Improving Pre-training Data Quality: Addressing the challenge of MLM on non-repetitive sequences and exploring alternative pre-training strategies are important future directions.

Summary updated

Abstract

Language models pretrained by self-supervised learning (SSL) have been widely utilized to study protein sequences, while few models were developed for genomic sequences and were limited to single species. Due to the lack of genomes from different species, these models cannot effectively leverage evolutionary information. In this study, we have developed SpliceBERT, a language model pretrained on primary ribonucleic acids (RNA) sequences from 72 vertebrates by masked language modeling, and applied it to sequence-based modeling of RNA splicing. Pretraining SpliceBERT on diverse species enables effective identification of evolutionarily conserved elements. Meanwhile, the learned hidden states and attention weights can characterize the biological properties of splice sites. As a result, SpliceBERT was shown effective on several downstream tasks: zero-shot prediction of variant effects on splicing, prediction of branchpoints in humans, and cross-species prediction of splice sites. Our study highlighted the importance of pretraining genomic language models on a diverse range of species and suggested that SSL is a promising approach to enhance our understanding of the regulatory logic underlying genomic sequences.
BibTeX
@article{chen2024self,
  author = {Ken Chen and Yue Zhou and Maolin Ding and Yu Wang and Zhixiang Ren and Yuedong Yang},
  title = {{Self-supervised learning on millions of primary RNA sequences from 72 vertebrates improves sequence-based RNA splicing prediction}},
  journal = {Briefings in Bioinformatics},
  year = {2024},
  url = {https://openreview.net/forum?id=kJE4MLfOZz},
}
Figure