Comprehensive Assessment of Isoform Detection Methods for Third-Generation Sequencing Data

Yaqi Su, Zhejian Yu, Siqian Jin, Zhipeng Ai, Ruihong Yuan, Xinyi Chen, Ziwei Xue, Yixin Guo, Di Chen, Hongqing Liang, Zuozhu Liu, Wanlu LiubioRxiv 2023 · DOI

TL;DR

This paper provides a comprehensive benchmark of eleven computational tools for identifying RNA isoforms from long-read sequencing data. It reveals that guided modes of StringTie2 and Bambu offer the best sensitivity and precision, respectively, offering crucial guidance for researchers in this rapidly evolving field.

What Problem Does This Paper Address?

Third-generation sequencing (TGS) technologies like PacBio and Oxford Nanopore have revolutionized transcriptomics by generating much longer sequencing reads. This advancement is crucial for accurately identifying alternative splicing (AS) events and quantifying gene isoforms, which are key to understanding biological complexity and disease. However, the rapid development of numerous computational tools for isoform detection has outpaced systematic comparative studies. Without such benchmarks, researchers struggle to choose the most appropriate tool for their specific needs, and developers lack clear direction for improving existing methods or creating new ones. This paper aims to fill that gap by rigorously evaluating the performance of leading TGS isoform detection tools under various simulated and real-world conditions.

Methods

The authors developed a novel simulator called YASIM (Yet Another SIMulator) to generate realistic TGS RNA-seq data. YASIM can simulate reads from different platforms (Nanopore and PacBio) and allows users to control various parameters, including sequencing depth, transcriptome complexity (number of isoforms per gene), read completeness, sequencing error rates, and reference annotation completeness. This simulated data, along with publicly available experimental TGS RNA-seq datasets and newly generated human embryonic stem cell (hESC) data, was used to benchmark eleven different isoform detection methods implemented in eight software tools. The evaluation focused on accuracy (precision and sensitivity), similarity of results between methods, downstream differential isoform usage (DIU) analysis, and computational performance (runtime and memory usage).

Datasets

The study utilized a combination of simulated and experimental datasets:

  • Simulated Data: Generated using the in-house YASIM simulator, mimicking Nanopore and PacBio sequencing technologies. These datasets were designed to cover a wide range of influencing factors such as sequencing depth (10X-70X), transcriptome complexity, read completeness (full-length to truncated), sequencing error rates, and reference annotation completeness.
  • Experimental Data:
    • Publicly available TGS RNA-seq datasets from four species: human, mouse, Drosophila melanogaster, and Caenorhabditis elegans.
  • Newly generated Nanopore TGS RNA-seq datasets from naïve and primed human embryonic stem cells (hESCs).
  • Previously published NGS RNA-seq datasets from naïve and primed hESCs under similar conditions.

Key Insights and Ideas

  • Guided vs. Unguided Performance: The study highlights that guided modes (using a reference annotation) of StringTie2 and Bambu generally outperform unguided modes, particularly in terms of sensitivity and precision, respectively. This suggests that leveraging existing genomic knowledge is still highly beneficial for accurate isoform detection.
  • The Importance of Read Completeness: The performance of most tools significantly degrades with incomplete reads, underscoring the need for high-quality, full-length sequencing data.
  • Error Correction Matters: Tools that incorporate an error correction step before isoform detection tend to be less sensitive to sequencing errors, though this can also lead to over-correction and filtering of true positives if the reference annotation is incomplete.
  • Transcriptome Complexity is a Challenge: As the number of isoforms per gene increases, detecting true positives becomes more difficult, and some tools tend to generate more false positives.
  • DIU Analysis is Sensitive to Isoform Detection: The choice of isoform detection tool can significantly impact downstream DIU analysis, with some tools identifying a large number of DIUs that are not detected by others, potentially due to differences in false positive rates.

Key Results

  • Top Performers: StringTie2 (guided) achieved the best sensitivity, while Bambu (guided) demonstrated the highest precision across various simulated scenarios. FLAIR and FLAMES also showed strong performance, especially with their additional functional modules.
  • Sequencing Depth: Increasing sequencing depth generally improved sensitivity for most methods, but did not consistently improve precision, possibly due to the long read lengths of TGS data.
  • Read Accuracy: Higher read accuracy generally led to improved precision, with PacBio CCS reads (highly accurate) showing better performance than other error-prone reads.
  • Annotation Completeness: All methods showed improved performance with increased reference annotation completeness, though the impact varied between tools.
  • Computational Performance: StringTie2 (both modes) was found to be the fastest and most memory-efficient, with excellent scalability. FLAMES, FLAIR, and Bambu also exhibited good computational performance.

Limitations and Future Work

The authors acknowledge that benchmarking results from simulated data may not fully capture the complexity of real-world data, and other factors like GC content were not analyzed. They suggest that future work could involve benchmarking new tools like ESPRESSO and further investigating the impact of various factors on isoform detection. The study also highlights the potential for long-read RNA-seq data to improve the identification of transposable elements (TEs) and their structures, suggesting this as a promising area for future research. The comprehensive evaluation provides a valuable resource for selecting appropriate tools and guiding the development of future isoform detection algorithms.

Summary updated

Abstract

Abstract The advancement of Third-Generation Sequencing (TGS) techniques has significantly increased the length of sequencing to several kilobases, thereby facilitating the identification of alternative splicing (AS) events and isoform expressions. Recently, numerous computational methods for isoform detection using long-read sequencing data have been developed. However, there is lack of prior comparative studies that systemically evaluates the performance of these software tools, implemented with different algorithms, under various simulations that encompass potential influencing factors. In this study, we conducted a benchmarking analysis of eleven methods implemented in eight computational tools capable of identifying isoform structures from TGS RNA sequencing data. We evaluated their performances using simulated data, which represented diverse sequencing platforms generated by an in-house simulator, as well as experimental data. Our comprehensive results demonstrate the guided mode of StringTie2 and Bambu achieved the best performance in sensitivity and precision, respectively. This study provides valuable guidance for future research on AS analysis and the ongoing improvement of tools for isoform detection using TGS data.
BibTeX
@misc{su2023comprehensive,
  author = {Yaqi Su and Zhejian Yu and Siqian Jin and Zhipeng Ai and Ruihong Yuan and Xinyi Chen and Ziwei Xue and Yixin Guo and Di Chen and Hongqing Liang and Zuozhu Liu and Wanlu Liu},
  title = {{Comprehensive Assessment of Isoform Detection Methods for Third-Generation Sequencing Data}},
  year = {2023},
  doi = {10.1101/2023.08.03.551905},
  howpublished = {bioRxiv},
}
Figure