Cytometry Masked Autoencoder: An Accurate and Interpretable Automated Immunophenotyper

Jaesik Kim, Matei Ionita, Matthew Lee, Michelle L. McKeague, Ajinkya Pattekar, Mark M. Painter, Joost Wagenaar, Van Truong, Dylan T. Norton, Divij Mathew, Yonghyun Nam, Sokratis A. Apostolidis, Cynthia Clendenin, Patryk Orzechowski, Sang‐Hyuk Jung, Jakob Woerner, C.a.g. Ittner, Alexandra P. Turner, Mika Esperanza, Thomas Dunn, Nilam S. Mangalmurti, John P. Reilly, Nuala J. Meyer, Carolyn S. Calfee, Kathleen D. Liu, Michael A. Matthy, Lamorna Brown Swigart, Ellen L. Burnham, Jeffrey McKeehan, Sheetal Gandotra, Derek W. Russel, Kevin W. Gibbs, Karl W. Thomas, Harsh Barot, Allison R. Greenplate, E. John Wherry, Dokyoon KimbioRxiv 2024 · DOI

TL;DR

This paper introduces cyMAE, a novel "Cytometry Masked Autoencoder," that automates the complex and time-consuming process of immunophenotyping using single-cell cytometry data. By leveraging a self-supervised pre-training approach inspired by natural language processing and computer vision, cyMAE achieves accurate cell type annotation and imputation, while also offering interpretable insights into marker importance, outperforming traditional manual gating and clustering methods.

What Problem Does This Paper Address?

High-throughput single-cell cytometry, like flow and mass cytometry, generates massive datasets crucial for understanding the immune system's role in health and disease. However, analyzing this data, particularly for cell type identification (immunophenotyping), has traditionally relied on manual gating. This process involves experts manually drawing boundaries on plots of protein marker expression, which is:

  • Time-consuming and labor-intensive: Especially for panels with many markers.
  • Subjective and prone to bias: Results can vary between analysts and depend on prior knowledge.
  • Difficult to replicate: Inconsistent gating strategies across studies hinder comparisons.

While automated clustering methods exist, they often struggle to differentiate biological variability from technical noise and can lead to inconsistent results across different datasets. This paper aims to bridge this gap by providing an accurate, robust, interpretable, and scalable automated solution for immunophenotyping.

Methods

The core of cyMAE is a Masked Autoencoder (MAE) architecture, adapted for cytometry data. It employs a two-phase training process:

  1. Self-supervised Pre-training (Masked Cytometry Modelling - MCM):
    • The model takes single-cell cytometry data (protein expression levels for millions of cells) and randomly masks a portion of the protein expression values.
  2. It then uses an encoder (based on transformer blocks) to learn latent representations of the unmasked protein expressions.
  3. A decoder attempts to reconstruct the masked protein expression values using these latent representations.
  4. This phase allows the model to learn complex relationships and dependencies between protein markers without any prior knowledge of cell identity. This is analogous to how models like BERT learn language by predicting masked words.
  5. Supervised Fine-tuning:
    • The pre-trained encoder is then used to generate robust cell representations.
  6. These representations are fed into a linear layer for specific downstream tasks, such as cell type annotation or imputation, using labeled data.
  7. This fine-tuning step adapts the general knowledge learned during pre-training to specific analytical goals.

Key novelties include the application of the MAE paradigm to cytometry data and the development of MCM for learning protein marker relationships. The model also incorporates attention mechanisms, which are crucial for interpretability.

Datasets

The study utilized data from three distinct COVID-19 studies conducted at the University of Pennsylvania:

  • Vaccine dataset: Longitudinal data from 37 healthy adults before and after SARS-CoV-2 mRNA vaccination (36.7M cells). Used for fine-tuning and internal testing.
  • Acute2020 dataset: Samples from patients hospitalized with acute SARS-CoV-2 infection and healthy controls (6.5M cells). Used for pre-training and external validation.
  • Acute2021 dataset: Longitudinal samples from patients with acute SARS-CoV-2 infection (11.9M cells). Used for external validation.

All datasets used a 30-marker panel. The study also mentions using a subset of the Vaccine dataset for training and evaluation, with the remaining for validation and testing.

Key Insights and Ideas

  • Self-supervised learning is powerful for cytometry: The MAE approach, by learning from unlabeled data, significantly reduces the reliance on expensive and time-consuming manual annotation.
  • Interpretable predictions: The attention mechanism within the transformer architecture allows researchers to understand which protein markers are most important for predicting cell types or imputing expression values, offering biological insights.
  • Robustness to technical variation: cyMAE demonstrated consistent performance across different datasets, even those with different collection and processing protocols, highlighting its generalizability.
  • Uncovering hidden information: cyMAE's performance in subject status prediction suggests it can capture critical information that might be missed by manual gating.
  • Few-shot learning capability: Once pre-trained, cyMAE can effectively learn new cell types with very limited labeled data, making it adaptable to new research questions.

Key Results

  • Cell Type Annotation: cyMAE achieved high Balanced Accuracy (Bacc) of 93.1% on the internal test set and 82.5% and 81.0% on external validation sets, outperforming other supervised methods (GBDT, DNN, CNN) and unsupervised clustering (FlowSOM). It showed particular strength in accurately predicting rare cell types.
  • Regression Imputation: cyMAE outperformed Infinity Flow (a gradient boosting model) in imputing masked marker expressions for five out of seven markers, achieving R-squared values up to 0.6. This suggests it can infer T and B cell memory states even without direct markers.
  • Subject Status Prediction: cyMAE achieved AUROC scores of 0.987 for COVID-19 patient classification, 0.669 for secondary immune response prediction, and 0.861 for pre- vs. post-treatment status. These results were comparable to or better than manual gating and FlowSOM, suggesting it captures critical information overlooked by traditional methods.
  • Computational Efficiency: Pre-training took approximately 10 days on multiple GPUs, while inference was fast (e.g., 1.2 hours for 7.3M cells on a single GPU).

Limitations and Future Work

  • Data Modality: The current model was trained on CyTOF data. While it shows promise, direct application to flow cytometry data might require further adaptation due to inherent technical differences. The authors suggest pre-training a model from scratch on flow cytometry data could be a viable approach.
  • Ground Truth Assumption: The study assumes manual gating provides the ground truth. However, the authors acknowledge that manual gating can be subjective and may miss information, as suggested by cyMAE's superior performance in some prediction tasks.
  • Fixed Marker Panels: The current model is trained for fixed marker panels. Future work should aim to make it robust to different panel sizes and subsets of markers.
  • Broader Pre-training: The authors suggest future research should expand pre-training to a broader array of data from multiple studies and diverse subject phenotypes to further enhance the model's power and robustness. This could lead to more accurate predictions and a deeper understanding of protein functions.

Summary updated

Abstract

Abstract High-throughput single-cell cytometry data are crucial for understanding involvement of immune system in diseases and responses to treatment. Traditional methods for annotating cytometry data, specifically manual gating and clustering, face challenges in scalability, robustness, and accuracy. In this study, we propose a cytometry masked autoencoder (cyMAE), which offers an automated solution for immunophenotyping tasks including cell type annotation. The cyMAE model is designed to uphold user-defined cell type definitions, thereby facilitating easier interpretation and cross-study comparisons. The cyMAE model operates on a pre-train and fine-tune approach. In the pre-training phase, cyMAE employs Masked Cytometry Modelling (MCM) to learn relationships between protein markers in immune cells solely based on protein expression, without relying on prior information such as cell identity and cell type-specific marker proteins. Subsequently, the pre-trained cyMAE is fine-tuned on multiple specialized tasks via task-specific supervised learning. The pre-trained cyMAE addresses the shortcomings of manual gating and clustering methods by providing accurate and interpretable predictions. Through validation across multiple cohorts, we demonstrate that cyMAE effectively identifies co-occurrence patterns of bound labeled antibodies, delivers accurate and interpretable cellular immunophenotyping, and improves the prediction of subject metadata status. Specifically, we evaluated cyMAE for cell type annotation and imputation at the cellular-level and SARS-CoV-2 infection prediction, secondary immune response prediction against COVID-19, and prediction of the infection stage in COVID-19 progression at the subject-level. The introduction of cyMAE marks a significant step forward in immunology research, particularly in large-scale and high-throughput human immune profiling. This approach offers new possibilities for predicting and interpreting cellular-level and subject-level phenotypes in both health and disease.
BibTeX
@misc{kim2024cytometry,
  author = {Jaesik Kim and Matei Ionita and Matthew Lee and Michelle L. McKeague and Ajinkya Pattekar and Mark M. Painter and Joost Wagenaar and Van Truong and Dylan T. Norton and Divij Mathew and Yonghyun Nam and Sokratis A. Apostolidis and Cynthia Clendenin and Patryk Orzechowski and Sang‐Hyuk Jung and Jakob Woerner and C.A.G. Ittner and Alexandra P. Turner and MIKA ESPERANZA and Thomas Dunn and Nilam S. Mangalmurti and John P. Reilly and Nuala J. Meyer and Carolyn S. Calfee and Kathleen D. Liu and Michael A. Matthy and Lamorna Brown Swigart and Ellen L. Burnham and Jeffrey McKeehan and Sheetal Gandotra and Derek W. Russel and Kevin W. Gibbs and Karl W. Thomas and Harsh Barot and Allison R. Greenplate and E. John Wherry and Dokyoon Kim},
  title = {{Cytometry Masked Autoencoder: An Accurate and Interpretable Automated Immunophenotyper}},
  year = {2024},
  doi = {10.1101/2024.02.13.580114},
  howpublished = {bioRxiv},
}
Figure