scROSHI: robust supervised hierarchical identification of single cells

Michael Prummer, Anne Bertolini, Lars Bosshard, Florian Barkmann, Josephine Yates, Valentina Boeva, Ruedi Aebersold, Melike Ak, Faisal Alquaddoomi, Jonas Albinus, Ilaria Alborelli, Sonali Andani, Per-Olof Attinger, Marina Bacac, Daniel Baumhoer, Beatrice Beck‐Schimmer, Niko Beerenwinkel, Christian Beisel, Lara Bernasconi, Bernd Bodenmiller, Ximena Bonilla, Byron Calgua, Ruben Casanova, Stéphane Chevrier, Natalia Chicherova, Maya D’Costa, Esther Danenberg, Natalie J. Davidson, Monica-Andreea Drăgan, Reinhard Dummer, Stefanie Engler, Martin Erkens, Katja Eschbach, Cinzia Esposito, André Fedier, Pedro Ferreira, Joanna Ficek, Anja Frei, Bruno S. Frey, Sandra Goetze, Linda Grob, Gabriele Gut, Detlef Günther, Martina Haberecker, Pirmin Haeuptle, Viola Heinzelmann‐Schwarz, Sylvia Herter, René Holtackers, Tamara Huesser, Anja Irmisch, Francis Jacob, Alice K. Jacobs, Tim M. Jaeger, Katharina Jahn, Alva Rani James, Philip Jermann, André Kahles, Abdullah Kahraman, Viktor H. Koelzer, Werner Kuebler, Jack Kuipers, Christian P. Kunze, Christian Kurzeder, Kjong-Van Lehmann, Mitchell Levesque, Sebastian Lugert, Gerd Maass, Markus G. Manz, Philipp Markolin, Julien Mena, Ulrike Menzel, Julian M. Metzler, Nicola Miglino, Emanuela S. Milani, Holger Moch, Simone Muenst, Riccardo Murri, Charlotte K.y. Ng, Stefan Nicolet, Marta Nowak, Patrick G. A. Pedrioli, Lucas Pelkmans, Salvatore Piscuoglio, Mathilde Ritter, Christian Rommel, María L. Rosano-González, Gunnar Rätsch, Natascha Santacroce, Jacobo Sarabia del Castillo, Ramona Schlenker, Petra Schwalie, Severin Schwan, Tobias Schär, Gabriela Senti, Franziska Singer, Sujana Sivapatham, Berend SnijderNAR 2022 · DOI

TL;DR

This paper introduces scROSHI, a novel method for identifying cell types in single-cell RNA sequencing data that bypasses the need for labeled training data. By leveraging pre-defined cell type-specific gene lists and a hierarchical classification approach, scROSHI achieves robust and accurate cell typing, outperforming existing methods, especially in scenarios with limited or diverse training data.

What Problem Does This Paper Address?

Single-cell transcriptomics has revolutionized biological research, enabling the creation of detailed "atlases" of cells within tissues. A crucial step in this process is cell type annotation – identifying what each cell is. Traditionally, this relies on machine learning models trained on large datasets with known cell type labels. However, this approach faces several challenges:

  • Data Scarcity: In early-stage research or when studying novel biological systems, labeled training data is often unavailable or insufficient.
  • Overfitting: Training and testing models on data from the same experimental conditions can lead to an overestimation of performance, as models learn condition-specific biases rather than generalizable features. This is particularly problematic when applying models to data from different experiments or labs.
  • Complexity of Cell Types: Distinguishing between closely related cell types (e.g., developing or precursor cells) is difficult, and the sheer number of potential cell types can increase misclassification rates.

The authors aim to overcome these limitations by developing a method that is less reliant on extensive, perfectly matched training data and can better handle biological complexity.

Methods

scROSHI (robust supervised hierarchical identification of single cells) is built on three core components:

  1. Cell Type-Specific Gene Lists: Instead of learning features from scratch, scROSHI utilizes pre-existing, validated lists of genes known to be highly expressed in specific cell types. The key is that these lists are based on gene names, not their expression levels, making the method less susceptible to variations in experimental conditions.
  2. Hierarchical Classification: scROSHI doesn't try to classify cells into all possible types at once. Instead, it follows a "family tree" approach, first assigning cells to broad "major types" and then progressively refining these assignments to more specific "subtypes" within those branches. This reduces the complexity at each step and helps differentiate closely related cell types.
  3. Quantitative and Robust Scoring: For each cell and each candidate cell type, scROSHI compares the expression of the cell type-specific genes against the expression of genes from other cell types. It uses the Mann-Whitney rank sum test to compare the ranks of gene expression, providing a robust score that is less sensitive to outliers and doesn't require data normalization. The negative log of the p-value from this test serves as a measure of how well a cell matches a given cell type.

scROSHI also incorporates "unknown" and "uncertain" labels to explicitly account for classification uncertainty, flagging cells that may represent novel types or require further investigation.

Datasets

The study utilized three publicly available peripheral blood mononuclear cell (PBMC) datasets for benchmarking:

  • Zheng_sorted set: An adult human dataset where cell types were pre-sorted.
  • Adult set: An adult human dataset.
  • Newborn set: A newborn human dataset.

These datasets were chosen to represent similar cell type compositions but differ in experimental settings and donor age, allowing for evaluation of inter-dataset performance. The authors also mention using curated gene lists from previous studies, such as the immune cell type gene lists (Im22) and resources for single-cell melanoma data.

Key Insights and Ideas

  • Leveraging Prior Knowledge is Powerful: The success of scROSHI highlights that well-curated, prior knowledge (in the form of gene lists) can be more robust than learning from scratch, especially when dealing with data variability.
  • Hierarchy Reduces Complexity: The hierarchical approach is a smart way to tackle the combinatorial explosion of cell types. By breaking down the problem, it makes classification more tractable and accurate, particularly for distinguishing similar cell types.
  • Robustness to Data Variability: By relying on gene ranks rather than absolute expression values, scROSHI is inherently more resistant to batch effects and variations in experimental protocols, a major advantage in multi-dataset analyses.
  • Explicit Uncertainty Handling: The "unknown" and "uncertain" labels are not just placeholders; they are crucial for identifying potential novel cell types and flagging cells that don't fit neatly into predefined categories, guiding further research.

Key Results

  • Superior Performance in Challenging Scenarios: scROSHI consistently outperformed supervised methods (SVM, Random Forest, GARNETT) when trained on data different from the test set (inter-dataset evaluation). This was particularly evident when the training and test datasets were diverse.
  • High Accuracy: For intra-dataset evaluation (training and testing on the same data), scROSHI performed comparably to supervised methods. For major cell types, accuracies ranged from 0.715 to 0.952 across datasets.
  • Low "Unknown" Cell Fraction: Even in challenging datasets like the Zheng set, scROSHI maintained a low percentage of "unknown" cells (below 10%), whereas ML-based methods often had significantly higher fractions, indicating they couldn't confidently assign labels.
  • Efficiency: The method is computationally efficient, classifying 2000 cells with 3368 genes into 11 cell types in under 35 seconds using 6 GB RAM on a standard laptop.

Limitations and Future Work

  • Dependence on Gene Lists: The performance of scROSHI is directly dependent on the quality and completeness of the provided cell type-specific gene lists. If a relevant cell type is missing from the lists, it may be misclassified or labeled as "unknown."
  • Future Work: The authors suggest that scROSHI is applicable to various data modalities and organisms, provided appropriate gene lists are available. They also note that while the current implementation is for tumor samples, it has no inherent limitations for other applications. The ability to identify "unexpected" cell types by removing them from the candidate list is a promising direction for discovering novel cell populations.

Summary updated

Abstract

Identifying cell types based on expression profiles is a pillar of single cell analysis. Existing machine-learning methods identify predictive features from annotated training data, which are often not available in early-stage studies. This can lead to overfitting and inferior performance when applied to new data. To address these challenges we present scROSHI, which utilizes previously obtained cell type-specific gene lists and does not require training or the existence of annotated data. By respecting the hierarchical nature of cell type relationships and assigning cells consecutively to more specialized identities, excellent prediction performance is achieved. In a benchmark based on publicly available PBMC data sets, scROSHI outperforms competing methods when training data are limited or the diversity between experiments is large.
BibTeX
@inproceedings{prummer2022scroshi,
  author = {Michael Prummer and Anne Bertolini and Lars Bosshard and Florian Barkmann and Josephine Yates and Valentina Boeva and Ruedi Aebersold and Melike Ak and Faisal Alquaddoomi and Jonas Albinus and Ilaria Alborelli and Sonali Andani and Per-Olof Attinger and Marina Bacac and Daniel Baumhoer and Beatrice Beck‐Schimmer and Niko Beerenwinkel and Christian Beisel and Lara Bernasconi and Anne Bertolini and Bernd Bodenmiller and Ximena Bonilla and Lars Bosshard and Byron Calgua and Ruben Casanova and Stéphane Chevrier and Natalia Chicherova and Maya D’Costa and Esther Danenberg and Natalie J. Davidson and Monica-Andreea Drăgan and Reinhard Dummer and Stefanie Engler and Martin Erkens and Katja Eschbach and Cinzia Esposito and André Fedier and Pedro Ferreira and Joanna Ficek and Anja Frei and Bruno S. Frey and Sandra Goetze and Linda Grob and Gabriele Gut and Detlef Günther and Martina Haberecker and Pirmin Haeuptle and Viola Heinzelmann‐Schwarz and Sylvia Herter and René Holtackers and Tamara Huesser and Anja Irmisch and Francis Jacob and Alice K. Jacobs and Tim M. Jaeger and Katharina Jahn and Alva Rani James and Philip Jermann and André Kahles and Abdullah Kahraman and Viktor H. Koelzer and Werner Kuebler and Jack Kuipers and Christian P. Kunze and Christian Kurzeder and Kjong-Van Lehmann and Mitchell Levesque and Sebastian Lugert and Gerd Maass and Markus G. Manz and Philipp Markolin and Julien Mena and Ulrike Menzel and Julian M. Metzler and Nicola Miglino and Emanuela S. Milani and Holger Moch and Simone Muenst and Riccardo Murri and Charlotte K.Y. Ng and Stefan Nicolet and Marta Nowak and Patrick G. A. Pedrioli and Lucas Pelkmans and Salvatore Piscuoglio and Michael Prummer and Mathilde Ritter and Christian Rommel and María L. Rosano-González and Gunnar Rätsch and Natascha Santacroce and Jacobo Sarabia del Castillo and Ramona Schlenker and Petra Schwalie and Severin Schwan and Tobias Schär and Gabriela Senti and Franziska Singer and Sujana Sivapatham and Berend Snijder},
  title = {{scROSHI: robust supervised hierarchical identification of single cells}},
  booktitle = {NAR},
  year = {2022},
  doi = {10.1093/nargab/lqad058},
}
Figure