TL;DR
This paper benchmarks five popular tools for automatically annotating cell types in single-cell ATAC-seq (scATAC-seq) data. They found that "Bridge integration," which leverages multimodal data, generally performed best across various metrics, while "Conos" was the most efficient but least accurate.
What Problem Does This Paper Address?
Single-cell ATAC-seq (scATAC-seq) is a powerful technique for understanding gene regulation by mapping open chromatin regions. However, interpreting these maps requires knowing which cell type each profile belongs to. While many tools exist for annotating single-cell RNA-seq (scRNA-seq) data, adapting these or developing new methods for scATAC-seq is challenging. Most existing scATAC-seq annotation tools rely on transferring labels from annotated scRNA-seq data. This paper addresses the critical need for a comprehensive evaluation of these scATAC-seq annotation tools to understand their performance, scalability, and robustness across different biological contexts and data conditions.
Methods
The authors evaluated five scATAC-seq annotation methods: Conos, Seurat v3, scGCN, scJoint, and Bridge integration. They employed a multi-faceted benchmarking approach:
- Accuracy Metrics: They used overall accuracy, weighted accuracy (which accounts for cell type similarity), and macro F1 scores. For ATAC-specific cell types (those not present in the reference scRNA-seq data), they introduced metrics like weighted accuracy and F1 of entropy and enrichment.
- Scalability Metrics: Running time and peak memory usage were recorded to assess efficiency.
- Experimental Scenarios: They systematically tested method performance under varying conditions:
- Different Tissues: Using datasets from mouse and human tissues (lung, brain, kidney, PBMC, BMMC).
- Data Size: Varying the number of cells in scRNA-seq and multimodal datasets.
- Mislabeling Proportion: Introducing artificial errors in scRNA-seq labels.
- Sequencing Depth: Downsampling data to simulate different sequencing depths.
- ATAC-Specific Cell Types: Artificially removing cell types from the reference scRNA-seq data to simulate novel cell types.
Datasets
The study utilized publicly available single-cell datasets from various mouse and human tissues:
- Mouse Lung: Both 10x Chromium (droplet-based) and Smart-seq2 (FACS-based) scRNA-seq data.
- Mouse Brain: scRNA-seq and multimodal data (SNARE-seq and SHARE-seq).
- Mouse Kidney: scRNA-seq and multimodal data.
- Human Peripheral Blood Mononuclear Cells (PBMC): scRNA-seq and multimodal data.
- Human Bone Marrow Mononuclear Cells (BMMC): A large, multimodal dataset serving as the primary benchmark for detailed experiments due to its comprehensive annotations and diverse cell types.
The BMMC dataset was particularly valuable as it allowed for controlled experiments by manually separating donors into unimodal RNA, unimodal ATAC, and multimodal groups.
Key Insights and Ideas
- Multimodal Data is Key for Robustness: Bridge integration, the only method requiring multimodal data, consistently showed strong performance and robustness across various challenging conditions, highlighting the benefit of integrating information from multiple modalities.
- Accuracy vs. Efficiency Trade-off: There's a clear trade-off. Conos is the fastest and most memory-efficient but performs the worst in accuracy. scGCN is computationally intensive but offers moderate accuracy.
- Handling ATAC-Specific Cell Types is Crucial: Methods differ significantly in their ability to handle cell types unique to ATAC-seq. scJoint tends to assign these to similar known cell types, which can be useful but might miss true novelty. scGCN and Bridge integration show better performance in distinguishing these unique cell types.
- Data Quality Matters: Performance degrades with lower sequencing depth and higher mislabeling rates, but Bridge integration shows remarkable resilience, especially to mislabeling.
- Cell Type Similarity is Important: Weighted accuracy, which considers cell type similarity, reveals different performance rankings than simple accuracy, with scJoint often excelling in this metric.
Key Results
- Overall Best Performer: Bridge integration was generally the top performer in terms of overall accuracy and F1 (macro), and the second-best for weighted accuracy. It was also robust to changes in data size, mislabeling rate, and sequencing depth.
- Most Efficient: Conos was the most time and memory efficient but had the lowest prediction accuracy.
- scJoint's Strengths and Weaknesses: scJoint performed well on simpler datasets but struggled with complex ones with deep annotations. It excelled in weighted accuracy, indicating good handling of cell type similarity.
- scGCN and Seurat v3: Showed moderate performance. scGCN was the most time-consuming and performed poorly on ATAC-specific cell types.
- Data Scale Sensitivity: Most methods stabilized in performance when the number of RNA cells exceeded 3k.
- Mislabeling Impact: Performance decreased significantly for most methods when mislabeling proportion exceeded 50%, with Bridge integration maintaining accuracy even at 70% mislabeling.
- Sequencing Depth: All methods were sensitive to low sequencing depth, but Bridge integration was most sensitive to ATAC and multimodal data depth, and least sensitive to RNA data depth.
Limitations and Future Work
- Tissue and Technology Specificity: The authors acknowledge that their conclusions might be specific to the tissues and technologies used. Future work could explore a wider range of tissues and sequencing platforms.
- Coarse Cell Type Granularity: The cell type annotations used were relatively coarse. Finer annotations could reveal different performance characteristics of the methods.
- Recommendations for Users:
- For users with available multimodal data (10x), Bridge integration is recommended.
- Otherwise, scJoint is a strong contender, with the caveat that it may misclassify ATAC-specific cell types.
- For users prioritizing the identification of ATAC-specific cell types, a two-step approach using scGCN or Seurat v3 followed by manual annotation might be beneficial.
Summary updated