A Comprehensive Benchmarking Study on Computational Tools for Cross-omics Label Transfer from Single-cell RNA to ATAC Data

Yuge Wang, Hongyu ZhaobioRxiv 2024 · DOI

TL;DR

This paper benchmarks 27 computational tools for transferring cell type labels from single-cell RNA sequencing (scRNA-seq) to single-cell ATAC sequencing (scATAC-seq) data. The study reveals that methods leveraging paired data and ATAC peak information perform best, while data imbalance and cross-omics dissimilarity significantly hinder accuracy. The findings offer crucial guidance for selecting and developing robust tools for cross-omics cell type annotation.

What Problem Does This Paper Address?

Accurately annotating cell types is fundamental to understanding the cellular composition of complex tissues. While scRNA-seq has been widely used for this purpose, scATAC-seq is increasingly employed to study regulatory genomic regions. However, annotating scATAC-seq data is challenging, with limited tools specifically designed for it. Many existing methods focus on label transfer within scRNA-seq or joint embedding of both modalities, which can then be used for label transfer. This paper addresses the need for a comprehensive evaluation of these tools to guide researchers in selecting the most effective methods for cross-omics label transfer from scRNA-seq to scATAC-seq.

Methods

The study evaluated 27 computational tools, categorizing them into those designed for direct label transfer and those for joint embedding. For joint embedding methods, a k-nearest neighbor (kNN) classifier was trained on the resulting latent space for label transfer. The authors also included kNN and random classifiers as baseline competitors. The performance was assessed using various accuracy metrics for common and ATAC-specific cell types, as well as scalability metrics like running time and peak memory usage.

Datasets

The study utilized a diverse collection of 27 datasets from both human and mouse tissues. These datasets were divided into three groups:

  • Paired data: Where scRNA-seq and scATAC-seq were measured simultaneously from the same cell (9 datasets).
  • Unpaired data with corresponding paired data: Where scRNA-seq and scATAC-seq were measured separately from the same tissue, but paired data from the same species and tissue were available (9 datasets).
  • Unpaired data without corresponding paired data: Where scRNA-seq and scATAC-seq were measured separately from the same tissue, and no paired data was available (9 datasets).

The authors also conducted specific experiments to investigate the impact of data properties like imbalance and dissimilarity, and method properties like the use of paired data and semi-supervised training.

Key Insights and Ideas

  • Paired data is king (when available): The availability of high-quality paired scRNA-seq and scATAC-seq data significantly boosts the performance of label transfer methods, with Bridge and GLUE (multi) emerging as top performers in such scenarios.
  • ATAC peak information is crucial: Methods that utilize raw ATAC peak data, rather than solely relying on gene activity, generally achieve better cell type separation and accuracy.
  • Data imbalance is a major hurdle: Datasets with a high proportion of modality-specific cell types or significant discrepancies in common cell type compositions lead to poor performance across most methods.
  • Semi-supervised training isn't a universal win: Introducing semi-supervised learning can negatively impact performance for many methods, suggesting it's not a one-size-fits-all solution and its effectiveness is method-dependent.
  • Deep learning excels at scalability: Deep learning-based methods generally demonstrate better scalability in terms of running time and memory usage, especially when leveraging GPU and minibatch training.

Key Results

  • Best performers: When high-quality paired data was available, Bridge and GLUE (multi) were the top performers. In the absence of paired data, bindSC and GLUE achieved the highest overall prediction accuracy.
  • ATAC-specific cell annotation remains challenging: No method performed exceptionally well on ATAC-specific cell types. scGCN showed performance closest to a random classifier, while scJoint tended to misclassify unobserved cell types with high confidence.
  • Negative impacts: Data imbalance, cross-omics dissimilarity, data binarization, and semi-supervised training generally had negative impacts on model performance.
  • Scalability: Bridge and deep-learning-based algorithms like GLUE were found to be the most time and memory efficient.

Limitations and Future Work

The authors acknowledge that their study has limitations, including the restricted variety of tissues and sequencing technologies covered. They also note that cell type labels derived from original publications might be subjective.

For future directions, they suggest that an ideal method should:

  • Thoroughly address data imbalance.
  • Incorporate ATAC peak-level information effectively.
  • Properly leverage high-quality paired data when available.
  • Avoid over-normalization techniques like binarization.
  • Use semi-supervised training cautiously.
  • Incorporate prior knowledge, such as regulatory interactions, to inform relationships between modalities.

Summary updated

Abstract

Abstract With continuous progress of single-cell chromatin accessibility profiling techniques, scATAC-seq has become more commonly used in investigating regulatory genomic regions and their involvement in developmental, evolutionary, and disease-related processes. At the same time, accurate cell type annotation plays a crucial role in comprehending the cellular makeup of complex tissues and uncovering novel cell types. Unfortunately, the majority of existing methods primarily focus on label transfer within scRNA-seq datasets and only a limited number of approaches have been specifically developed for transferring labels from scRNA-seq to scATAC-seq data. Moreover, many methods have been published for the joint embedding of data from the two modalities, which can be used for label transfer by adding a classifier trained on the latent space. Given these available methods, this study presents a comprehensive benchmarking study evaluating 27 computational tools for scATAC-seq label annotations through tasks involving single-cell RNA and ATAC data from various human and mouse tissues. We found that when high quality paired data were available to transfer labels across unpaired data, Bridge and GLUE were the best performers; otherwise, bindSC and GLUE achieved the highest prediction accuracy overall. All these methods were able to use peak-level information instead of purely relying on the gene activities from scATAC-seq. Furthermore, we found that data imbalance, cross-omics dissimilarity on common cell types, data binarization, and the introduction of semi-supervised strategy usually had negative impacts on model performance. In terms of scalability, we found that the most time and memory efficient methods were Bridge and deep-learning-based algorithms like GLUE. Based on the results of this study, we provide several suggestions for future methodology development.
BibTeX
@misc{wang2024comprehensive,
  author = {Yuge Wang and Hongyu Zhao},
  title = {{A Comprehensive Benchmarking Study on Computational Tools for Cross-omics Label Transfer from Single-cell RNA to ATAC Data}},
  year = {2024},
  doi = {10.1101/2024.02.01.578507},
  howpublished = {bioRxiv},
}
Figure