IBRAP: Integrated Benchmarking Single-cell RNA-sequencing Analytical Pipeline

Connor H. Knight, Faraz Khan, Upkar S. Gill, Jun WangbioRxiv 2022 · DOI

TL;DR

Single-cell RNA sequencing (scRNA-seq) generates complex data requiring specialized analytical pipelines. This paper introduces IBRAP, a flexible and comprehensive R-based toolkit that integrates numerous analytical components and benchmarking metrics, allowing users to find optimal pipelines for their specific scRNA-seq data. IBRAP demonstrates the superiority of reference-based cell annotation for identifying cell types, especially rare ones.

What Problem Does This Paper Address?

The analysis of single-cell RNA sequencing (scRNA-seq) data is a complex undertaking. The high dimensionality and intricate nature of this data necessitate specialized expertise for accurate interpretation. A critical challenge lies in the sheer number of analytical steps involved (e.g., pre-processing, normalization, dimensionality reduction, integration, clustering), each with a multitude of available algorithms. These algorithms often have different underlying assumptions and can produce varying results depending on the data's characteristics. This vast landscape of choices makes it difficult for researchers to select the most appropriate analytical pipeline for their specific datasets, leading to a need for robust benchmarking and integrated tools.

Methods

IBRAP is presented as a wrapper package designed to address the complexity of scRNA-seq analysis. It offers a modular approach, allowing users to interchange various analytical components. Key features include:

  • Interchangeable Components: IBRAP incorporates multiple methods for crucial steps like normalization (4 methods), integration (4 methods), and clustering (7 methods). This allows for the creation of a vast number of unique pipeline combinations.
  • Benchmarking Metrics: The tool includes several benchmarking metrics to evaluate pipeline performance, such as Average Silhouette Width (ASW), Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). These metrics help users assess clustering quality and batch effect removal.
  • Rshiny Interface: An interactive Rshiny application, "IBRAP Exploration," is provided for easier visualization and exploration of results, including dimensionality reduction plots, benchmarking visualizations, and gene expression plots.
  • Supervised and Unsupervised Analysis: IBRAP supports both unsupervised clustering (standard approach) and supervised cell annotation using reference-based maps (e.g., SingleR), enabling a direct comparison of their effectiveness.

Datasets

The authors evaluated IBRAP using a combination of publicly available and simulated datasets:

  • Pancreas Datasets: 8 datasets from 4 studies, using 5 different technologies (CELseq, CELseq2, SmartSeq2, Fluidigmc1, InDrop), with known cell type labels.
  • Cell Line Datasets: 7 mixed cell line samples from a previous study, using technologies like 10x Genomics, Dropseq, and CELseq2.
  • Simulated Datasets: 4 datasets generated using the R package "Splatter," with varying numbers of cell types and cell distributions to mimic different biological scenarios.

Key Insights and Ideas

  • No One-Size-Fits-All Pipeline: The study strongly emphasizes that the optimal scRNA-seq analysis pipeline is highly dependent on the specific dataset, including its biological complexity and technical variations. This underscores the necessity of flexible tools like IBRAP.
  • Reference-Based Annotation Superiority: For cell type identification, particularly for distinguishing rare cell populations, reference-based (supervised) annotation methods demonstrated superior performance compared to unsupervised clustering. This is attributed to the prior knowledge embedded in the reference maps.
  • Importance of Careful Method Selection: The choice of normalization and integration methods significantly impacts downstream analysis results, including batch effect removal and cell type annotation accuracy. Careful selection is crucial for reliable outcomes.

Key Results

  • IBRAP's Comprehensive Nature: IBRAP integrates over 100 packages and dependencies, offering a broad range of scRNA-seq specific normalization, dimensionality reduction, and clustering methods, surpassing other existing pipelines in terms of component variety.
  • Pipeline Performance Varies: Benchmarking across pancreas, cell line, and simulated datasets revealed that different pipeline combinations yield varying results. For instance, pancreas samples generally benefited from graph-based clustering, while cell lines performed better with non-graph-based methods.
  • SCTransform and Scran Excel in Normalization: SCTransform and Scran were identified as top-performing normalization methods for individual and integrated analyses, respectively, in many scenarios.
  • Harmony and Seurat CCA for Integration: Harmony showed strong performance in cell type label annotation, while Seurat CCA was most effective at removing batch effects, though not always leading to the best clustering.
  • Supervised vs. Unsupervised: On average, supervised analysis resulted in significantly fewer failed clusters (1.6) compared to unsupervised analysis (5.9) when identifying cell types against ground truth labels.

Limitations and Future Work

  • Limitations:
    • IBRAP does not include FASTQ alignment to a reference genome.
  • It cannot infer RNA velocity from spliced/unspliced matrix counts.
  • It does not include differential abundance tools.
  • Maintaining package dependencies requires ongoing effort, though this is partially mitigated by its availability as a Docker image.
  • Future Work:
    • The authors suggest that IBRAP's components, particularly pipeline benchmarking and integration methods, can be further refined for scalability to reference and Atlas-sized projects.
  • The development of supervised clustering for analyzing new samples is highlighted as a part of IBRAP's utility.
  • The paper points out that high-quality reference maps are still lacking in many settings, indicating a need for further research in this area.

Summary updated

Abstract

Abstract Single-cell RNA-sequencing (scRNA-seq) is a powerful tool to study cellular heterogeneity. The high dimensional data generated from this technology are complex and require specialised expertise for analysis and interpretation. The core of scRNA-seq data analysis contains several key analytical steps, which include pre-processing, QC, normalisation, dimensionality reduction, integration, and clustering. Each step often has many algorithms developed with varied underlying assumptions and implications. With such a diverse choice of tools available, benchmarking analyses have compared their performances and demonstrated that tools differentially operate according to the data types and complexity. Here, we present Integrated Benchmarking scRNA-seq Analytical Pipeline (IBRAP) – a tool, which contains a range of analytical components that can be interchanged throughout the pipeline alongside multiple benchmarking metrics that enables users to compare results and determine the optimal pipeline combinations for their data. We apply IBRAP to single and multi-sample integration analysis using pancreas, cell line and simulated data accompanied with ground truth cell labels, demonstrating the interchangeable and benchmarking functionality of IBRAP. Our results confirm that the optimal pipelines are dependant of individual samples and studies, further supporting the rationale and necessity of our tool. We then compare reference-based cell annotation with unsupervised analysis, both included in IBRAP, and demonstrate the superiority of the reference-based method in identifying robust major and minor cell types. Thus, IBRAP presents a valuable tool to integrate multiple samples and studies to create reference maps of normal and diseased tissues, facilitating novel biological discovery using the vast volume of scRNA-seq data available.
BibTeX
@misc{knight2022ibrap,
  author = {Connor H. Knight and Faraz Khan and Upkar S. Gill and Jun Wang},
  title = {{IBRAP: Integrated Benchmarking Single-cell RNA-sequencing Analytical Pipeline}},
  year = {2022},
  doi = {10.1101/2022.09.26.509481},
  howpublished = {bioRxiv},
}
Figure