Benchmarking Automated Cell Type Annotation Tools for Single-cell ATAC-seq Data

Yuge Wang, Xingzhi Sun, Hongyu Zhao

TL;DR

This paper benchmarks five popular tools for automatically annotating cell types in single-cell ATAC-seq (scATAC-seq) data. They found that "Bridge integration," which leverages multimodal data, generally performed best across various metrics, while "Conos" was the most efficient but least accurate.

What Problem Does This Paper Address?

Single-cell ATAC-seq (scATAC-seq) is a powerful technique for understanding gene regulation by mapping open chromatin regions. However, interpreting these maps requires knowing which cell type each profile belongs to. While many tools exist for annotating single-cell RNA-seq (scRNA-seq) data, adapting these or developing new methods for scATAC-seq is challenging. Most existing scATAC-seq annotation tools rely on transferring labels from annotated scRNA-seq data. This paper addresses the critical need for a comprehensive evaluation of these scATAC-seq annotation tools to understand their performance, scalability, and robustness across different biological contexts and data conditions.

Methods

The authors evaluated five scATAC-seq annotation methods: Conos, Seurat v3, scGCN, scJoint, and Bridge integration. They employed a multi-faceted benchmarking approach:

  • Accuracy Metrics: They used overall accuracy, weighted accuracy (which accounts for cell type similarity), and macro F1 scores. For ATAC-specific cell types (those not present in the reference scRNA-seq data), they introduced metrics like weighted accuracy and F1 of entropy and enrichment.
  • Scalability Metrics: Running time and peak memory usage were recorded to assess efficiency.
  • Experimental Scenarios: They systematically tested method performance under varying conditions:
  • Different Tissues: Using datasets from mouse and human tissues (lung, brain, kidney, PBMC, BMMC).
  • Data Size: Varying the number of cells in scRNA-seq and multimodal datasets.
  • Mislabeling Proportion: Introducing artificial errors in scRNA-seq labels.
  • Sequencing Depth: Downsampling data to simulate different sequencing depths.
  • ATAC-Specific Cell Types: Artificially removing cell types from the reference scRNA-seq data to simulate novel cell types.

Datasets

The study utilized publicly available single-cell datasets from various mouse and human tissues:

  • Mouse Lung: Both 10x Chromium (droplet-based) and Smart-seq2 (FACS-based) scRNA-seq data.
  • Mouse Brain: scRNA-seq and multimodal data (SNARE-seq and SHARE-seq).
  • Mouse Kidney: scRNA-seq and multimodal data.
  • Human Peripheral Blood Mononuclear Cells (PBMC): scRNA-seq and multimodal data.
  • Human Bone Marrow Mononuclear Cells (BMMC): A large, multimodal dataset serving as the primary benchmark for detailed experiments due to its comprehensive annotations and diverse cell types.

The BMMC dataset was particularly valuable as it allowed for controlled experiments by manually separating donors into unimodal RNA, unimodal ATAC, and multimodal groups.

Key Insights and Ideas

  • Multimodal Data is Key for Robustness: Bridge integration, the only method requiring multimodal data, consistently showed strong performance and robustness across various challenging conditions, highlighting the benefit of integrating information from multiple modalities.
  • Accuracy vs. Efficiency Trade-off: There's a clear trade-off. Conos is the fastest and most memory-efficient but performs the worst in accuracy. scGCN is computationally intensive but offers moderate accuracy.
  • Handling ATAC-Specific Cell Types is Crucial: Methods differ significantly in their ability to handle cell types unique to ATAC-seq. scJoint tends to assign these to similar known cell types, which can be useful but might miss true novelty. scGCN and Bridge integration show better performance in distinguishing these unique cell types.
  • Data Quality Matters: Performance degrades with lower sequencing depth and higher mislabeling rates, but Bridge integration shows remarkable resilience, especially to mislabeling.
  • Cell Type Similarity is Important: Weighted accuracy, which considers cell type similarity, reveals different performance rankings than simple accuracy, with scJoint often excelling in this metric.

Key Results

  • Overall Best Performer: Bridge integration was generally the top performer in terms of overall accuracy and F1 (macro), and the second-best for weighted accuracy. It was also robust to changes in data size, mislabeling rate, and sequencing depth.
  • Most Efficient: Conos was the most time and memory efficient but had the lowest prediction accuracy.
  • scJoint's Strengths and Weaknesses: scJoint performed well on simpler datasets but struggled with complex ones with deep annotations. It excelled in weighted accuracy, indicating good handling of cell type similarity.
  • scGCN and Seurat v3: Showed moderate performance. scGCN was the most time-consuming and performed poorly on ATAC-specific cell types.
  • Data Scale Sensitivity: Most methods stabilized in performance when the number of RNA cells exceeded 3k.
  • Mislabeling Impact: Performance decreased significantly for most methods when mislabeling proportion exceeded 50%, with Bridge integration maintaining accuracy even at 70% mislabeling.
  • Sequencing Depth: All methods were sensitive to low sequencing depth, but Bridge integration was most sensitive to ATAC and multimodal data depth, and least sensitive to RNA data depth.

Limitations and Future Work

  • Tissue and Technology Specificity: The authors acknowledge that their conclusions might be specific to the tissues and technologies used. Future work could explore a wider range of tissues and sequencing platforms.
  • Coarse Cell Type Granularity: The cell type annotations used were relatively coarse. Finer annotations could reveal different performance characteristics of the methods.
  • Recommendations for Users:
    • For users with available multimodal data (10x), Bridge integration is recommended.
  • Otherwise, scJoint is a strong contender, with the caveat that it may misclassify ATAC-specific cell types.
  • For users prioritizing the identification of ATAC-specific cell types, a two-step approach using scGCN or Seurat v3 followed by manual annotation might be beneficial.

Summary updated

Abstract

Abstract As single-cell chromatin accessibility profiling methods advance, scATAC-seq has become ever more important in the study of candidate regulatory genomic regions and their roles underlying developmental, evolutionary and disease processes. At the same time, cell type annotation is critical in understanding the cellular composition of complex tissues and identifying potential novel cell types. However, most existing methods that can perform automated cell type annotation are designed to transfer labels from an annotated scRNA-seq data set to another scRNA-seq data set, and it is not clear whether these methods are adaptable to annotate scATAC-seq data. Several methods have been recently proposed for label transfer from scRNA-seq data to scATAC-seq data, but there is a lack of benchmarking study on the performance of these methods. Here, we evaluated the performance of five scATAC-seq annotation methods on both their classification accuracy and scalability using publicly available single-cell datasets from mouse and human tissues including brain, lung, kidney, PBMC and BMMC. Using the BMMC data as basis, we further investigated the performance of these methods across different data sizes, mislabeling rates, sequencing depths and the number of cell types unique to scATAC-seq. Bridge integration, which is the only method that requires additional multimodal data and does not need gene activity calculation, was overall the best method and robust to changes in data size, mislabeling rate and sequencing depth. Conos was the most time and memory efficient method but performed the worst in terms of prediction accuracy. scJoint tended to assign cells to similar cell types and performed relatively poorly for complex datasets with deep annotations but performed better for datasets only with major label annotations. The performance of scGCN and Seurat v3 was moderate, but scGCN was the most time-consuming method and had the most similar performance to random classifiers for cell types unique to scATAC-seq.
BibTeX
@misc{wang2022benchmarking,
  author = {Yuge Wang and Xingzhi Sun and Hongyu Zhao},
  title = {{Benchmarking Automated Cell Type Annotation Tools for Single-cell ATAC-seq Data}},
  year = {2022},
  doi = {10.1101/2022.10.05.511014},
  howpublished = {bioRxiv},
}
Figure