SCTrans: Multi-scale scRNA-seq Sub-vector Completion Transformer for Gene-selective Cell Type Annotation

Lu Lin, Wen Xue, Xindian Wei, Wenjun Shen, Cheng Liu, Si Wu, Hau-San WongIJCAI 2024

Abstract

SCTrans is a novel Transformer-based framework for cell type annotation that shifts the focus from individual genes to multi-scale gene "sub-vectors." By combining masked sub-vector completion with a gene-selective mutual learning mechanism, it achieves state-of-the-art accuracy and provides biologically interpretable insights into which genes drive specific cell classifications.

Introduction

Single-cell RNA sequencing (scRNA-seq) is a powerful tool for understanding cellular diversity, but accurately labeling (annotating) cell types remains a challenge. Most existing machine learning methods treat gene expression as a simple vector where each gene is an independent feature. However, this approach suffers from three major limitations:

  1. Ignoring Interdependency: Genes function in complex co-expression networks rather than in isolation. Standard models often fail to capture these contextual relationships.
  2. Interpretability vs. Performance: While deep learning models like Transformers are highly accurate, they are often "black boxes." Identifying specific genes used for classification is crucial for biological research.
  3. Data Sparsity: scRNA-seq data is notoriously noisy and sparse, with many zero values that can mislead simple classification models.

SCTrans addresses these gaps by introducing a multi-scale sub-vector approach that treats sparsity as a learning opportunity rather than a hindrance.

Methodology

The SCTrans architecture is built around several innovative components designed to capture the underlying "grammar" of gene expression.

Detailed architecture of SCTrans, illustrating (a) sub-vector completion, (b) mutual learning between encoders, and (c) the classification module.

Multi-scale Tokenization and Completion

Instead of treating each gene as a single token, SCTrans partitions gene expression vectors into "sub-vectors" of varying sizes. This allows the model to capture patterns at different levels of biological granularity. Borrowing from the "Masked Autoencoder" (MAE) philosophy, the model randomly masks a majority of these sub-vectors during training and learns to reconstruct the missing parts based on the unmasked ones.

Gene-Selective Module and Mutual Learning

The framework employs two encoders: a main encoder ($E$) for broad representation learning and a lightweight gene-selection encoder ($E_S$). The selection encoder uses attention scores to identify which sub-vectors are most salient for classification. Through a mutual learning process, these two encoders refine each other's performance.

Conditional Contrastive Regularization

To ensure the model clearly distinguishes between cell types, a contrastive loss is applied. This pulls representations of cells of the same type closer together in the latent space while pushing different types further apart.

Experimental Results

SCTrans was evaluated on seven representative benchmark datasets covering various tissues, including PBMC (Zheng68K), pancreas, liver, and lung. These datasets demonstrate the model's flexibility across different sequencing protocols such as 10X Chromium and SMART-Seq2.

Performance and Robustness

SCTrans consistently outperformed state-of-the-art methods like scBERT and Seurat. On the difficult Zheng68K dataset, it achieved an accuracy of 0.817, significantly higher than scBERT’s 0.759. The model also demonstrated robustness to data imbalance and batch effects.

Confusion matrices comparing SCTrans and scBERT, highlighting the superior classification accuracy of SCTrans across various cell types (a-f).

Biological Interpretability

The attention scores generated by the gene-selective module often correlate with known marker genes for specific cell types. This validation suggests that SCTrans identifies true biological signals, making it a valuable tool for identifying new biomarkers.

Conclusion

SCTrans represents a significant step forward in scRNA-seq analysis by leveraging multi-scale sub-vectors and self-supervised completion. While computationally intensive, its ability to handle sparse data and provide interpretable results makes it a robust framework for cellular annotation. Future work will explore scaling the model to foundation-level datasets and integrating multi-omics data.

Summary updated

BibTeX
@inproceedings{lin2024sctrans,
  author = {Lu Lin and Wen Xue and Xindian Wei and Wenjun Shen and Cheng Liu and Si Wu and Hau-San Wong},
  title = {{SCTrans: Multi-scale scRNA-seq Sub-vector Completion Transformer for Gene-selective Cell Type Annotation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure