Scarf enables a highly memory-efficient analysis of large-scale single-cell genomics data

Parashar Dhapola, Johan Rodhe, Rasmus Olofzon, Thomas Bonald, Eva Erlandsson, Shamit Soneji, Göran KarlssonNature Communications 2022 · DOI

TL;DR

Researchers have developed "Scarf," a Python package that makes analyzing massive single-cell genomics datasets (millions of cells) feasible on standard laptops. Scarf achieves this through memory-efficient algorithms, enabling advanced analyses like subsampling, clustering, and data integration without requiring supercomputers.

What Problem Does This Paper Address?

Single-cell genomics is a rapidly advancing field, generating vast amounts of data from millions of cells. Analyzing this data requires significant computational resources, particularly memory (RAM), which often exceeds what's available on standard laptops or even mid-range servers. This "resource barrier" limits access for many biologists and bioinformaticians, hindering their ability to explore complex datasets, integrate different studies, or identify rare cell populations. Existing tools, while powerful, often struggle with memory scalability, forcing users to rely on specialized hardware or cloud computing.

Methods

Scarf is a modular Python package designed for memory-efficient single-cell data analysis. Its core innovations include:

  • Memory-Efficient Data Handling: Scarf processes large datasets by dividing them into smaller "chunks," compressing them, and storing them on disk. This "out-of-core" approach avoids loading the entire dataset into RAM. It utilizes the Zarr file format, which offers better support for parallel operations and remote data interactions compared to HDF5.
  • Incremental Algorithms: For computationally intensive steps like PCA and KNN, Scarf employs "out-of-core" or incremental implementations that process data in small chunks, further reducing memory demands.
  • Paris Hierarchical Clustering: A novel, scalable hierarchical clustering algorithm designed for millions of cells, which creates a dendrogram to represent cluster relationships at various granularities.
  • Topology-Assisted Cell Downsampling (TopACeDo): A subsampling algorithm that leverages the cell-cell neighborhood graph to create representative subsets of data, crucially preserving rare cell populations and lineage differentiation trajectories.
  • Reference-Anchored Mapping: An efficient KNN-based approach for integrating datasets by projecting query cells onto a reference manifold, minimizing batch effects and enabling co-embedding.

Datasets

The paper demonstrates Scarf's capabilities using several large-scale datasets:

  • scRNA-Seq Datasets: Ranging from 10K to 4 million cells, including datasets from human PBMCs and mouse brain.
  • scATAC-Seq Dataset: A large dataset of 720,613 cells from human fetal samples across 15 organs.
  • Pancreatic Cell Datasets: Multiple datasets from different labs and platforms (InDrop, CEL-Seq2, SMART-Seq2, C1-IFC SMARTer) used for multi-dataset integration.
  • Mouse Nervous System Atlas: A dataset of 146,093 cells used for mapping and integration benchmarks.

The paper also introduces a benchmark comparing Scarf's performance against existing tools like Scanpy and Seurat on these datasets.

Key Insights and Ideas

  • Memory is the Bottleneck, Not Just Compute: The authors highlight that memory capacity is the primary hurdle for analyzing large single-cell datasets, even more so than computational speed.
  • Laptop-Scale Atlas Analysis is Now Possible: Scarf democratizes access to large-scale genomics by enabling analysis on standard hardware, removing the need for expensive supercomputing resources.
  • Subsampling Preserves Biological Signal: The TopACeDo algorithm is a significant contribution, showing that intelligent subsampling can retain crucial biological information (rare cell types, differentiation paths) while drastically reducing data size for downstream analyses.
  • Data Integration Without Batch Effects: Scarf's reference-anchored mapping effectively integrates datasets from different sources, preserving the underlying biological structure and minimizing technical artifacts.
  • UMAP vs. SG-tSNE: The paper provides a comparative analysis of UMAP and SG-tSNE, suggesting that SG-tSNE might offer better visualization of heterogeneity and scalability for very large datasets, especially with more iterations.

Key Results

  • Memory Efficiency: Scarf consumed substantially less memory than Scanpy and Seurat. For a 4 million cell dataset, Scarf used less than 16 GB of RAM, while Scanpy failed to process it even with 200 GB available.
  • Compute Efficiency: Scarf processed a 4 million cell dataset in under 10 hours without exceeding 16GB of memory.
  • Subsampling Performance: Scarf's TopACeDo algorithm successfully preserved rare cell populations and lineage trajectories, outperforming random sampling in maintaining cell connectivity and enriching for smaller clusters. For example, it increased the proportion of cells from the smallest clusters by 8-16 fold, while random sampling showed minimal increase.
  • Data Integration Accuracy: Scarf accurately mapped and co-embedded datasets, with predicted cell types showing high accuracy, even for low-abundant populations. For instance, it achieved >99.8% accuracy in identifying alpha cells in pancreatic datasets.
  • Scalability: Scarf handled a >700K cell scATAC-Seq dataset in ~5.5 hours using less than 2.5 GB of RAM.

Limitations and Future Work

  • Subsampling Dependency: The TopACeDo subsampling algorithm requires a pre-computed neighborhood graph, which Scarf generates. This means it's not a standalone solution for tools that don't rely on such graphs.
  • Parameter Tuning: While Scarf aims to reduce the need for extensive parameter tuning, some steps (like UMAP and SG-tSNE) still benefit from it. The paper notes that SG-tSNE might offer better visualization with more iterations, suggesting further exploration of parameter optimization.
  • Future Work: The authors suggest that Scarf's modular design and efficient algorithms make it a strong foundation for future development, including further integration with other single-cell analysis tools and potentially expanding its multi-omics capabilities. The comparison of UMAP and SG-tSNE also opens avenues for further research into optimal visualization strategies for different dataset sizes and biological questions.

Summary updated

Abstract

As the scale of single-cell genomics experiments grows into the millions, the computational requirements to process this data are beyond the reach of many. Herein we present Scarf, a modularly designed Python package that seamlessly interoperates with other single-cell toolkits and allows for memory-efficient single-cell analysis of millions of cells on a laptop or low-cost devices like single-board computers. We demonstrate Scarf's memory and compute-time efficiency by applying it to the largest existing single-cell RNA-Seq and ATAC-Seq datasets. Scarf wraps memory-efficient implementations of a graph-based t-stochastic neighbour embedding and hierarchical clustering algorithm. Moreover, Scarf performs accurate reference-anchored mapping of datasets while maintaining memory efficiency. By implementing a subsampling algorithm, Scarf additionally has the capacity to generate representative sampling of cells from a given dataset wherein rare cell populations and lineage differentiation trajectories are conserved. Together, Scarf provides a framework wherein any researcher can perform advanced processing, subsampling, reanalysis, and integration of atlas-scale datasets on standard laptop computers. Scarf is available on Github: https://github.com/parashardhapola/scarf .
BibTeX
@inproceedings{dhapola2022scarf,
  author = {Parashar Dhapola and Johan Rodhe and Rasmus Olofzon and Thomas Bonald and Eva Erlandsson and Shamit Soneji and Göran Karlsson},
  title = {{Scarf enables a highly memory-efficient analysis of large-scale single-cell genomics data}},
  booktitle = {Nature Communications},
  year = {2022},
  doi = {10.1038/s41467-022-32097-3},
}
Figure