Abstract
Single-cell multiomic data, combining gene expression (scRNA-seq) and chromatin accessibility (scATAC-seq), provides a comprehensive view of cellular states and regulatory logic. However, integrating these high-dimensional, sparse, and noisy modalities remains a challenge. We present scMBERT, a BERT-based framework pre-trained on multiomic data using a novel modality prediction task. scMBERT achieves state-of-the-art performance in batch-effect correction and competitive accuracy in cell type annotation, demonstrating the power of cross-modal consistency in single-cell analysis.
Introduction
Single-cell RNA sequencing (scRNA-seq) has transformed our understanding of cellular diversity by profiling gene expression. Yet, RNA alone only captures the output of cellular processes. To understand the underlying regulatory mechanisms, researchers increasingly turn to multiomic assays like scATAC-seq, which measures the "regulome" via chromatin accessibility.
The integration of these modalities is difficult due to their extreme sparsity and high dimensionality. While Large Language Models (LLMs) like BERT and GPT have been adapted for single-cell data (e.g., scBERT, scGPT), most focus primarily on RNA or are constrained by fixed gene sets. scMBERT addresses these limitations by providing a flexible transformer architecture capable of integrating multiple cellular "views" simultaneously.
Methods
The scMBERT framework adapts the Bidirectional Encoder Representations from Transformers (BERT) architecture for multiomic sequences. Unlike models that process fixed gene sets, scMBERT handles variable inputs by processing only non-zero features, enhancing computational efficiency and cross-dataset flexibility.
The training process consists of two stages:
- Self-supervised Pre-training: The model learns biological representations from unlabeled data through two tasks:
- Masked Value Prediction (MVP): A standard BERT-style task where the model predicts masked expression or accessibility values.
- Data Modality Prediction (DMP): A novel task requiring the model to identify the source (RNA or ATAC) of a data segment. This serves as a regularizer to ensure cross-modal consistency.
- Supervised Fine-tuning: The pre-trained encoder is coupled with a classification head for tasks such as cell type annotation.
Experimental Setup
The model was pre-trained on the ENCODE 4 portal, encompassing 47 single-cell multiomic datasets across 13 tissue types. Evaluation was conducted using the NeurIPS BMMC (Bone Marrow Mononuclear Cells) dataset, containing approximately 69,000 cells from 12 donors. To assess generalizability, data from three donors were held out during training.
Results
scMBERT was benchmarked against leading single-cell models, scBERT and scGPT, focusing on cell type annotation accuracy, biological signal preservation (AvgBIO), and batch effect correction (AvgBATCH).
Key findings include:
- Accuracy: scMBERT (Multiome) achieved 75.56% accuracy, comparable to the 76.68% achieved by scGPT, a significantly larger generative model.
- Batch Correction: scMBERT significantly outperformed competitors with an AvgBATCH score of 0.9072 (vs. 0.8039 for scGPT and 0.8696 for scBERT). This indicates a superior ability to merge data across different donors and technical batches.
- Biological Representation: With an AvgBIO score of 0.6362, scMBERT demonstrated better preservation of distinct biological signals compared to scGPT (0.5272).
Discussion
The results underscore the advantages of multiomic integration. By incorporating both the transcriptome and the regulome, scMBERT gains a more robust understanding of cellular identity. The introduction of the DMP task effectively prevents overfitting to a single modality.
Future work will focus on scaling pre-training to larger multiomic datasets and extending the model's capabilities to tasks such as missing data imputation (e.g., predicting ATAC-seq from RNA-seq) and signal enhancement. Currently, the requirement for "unified open chromatin regions" necessitates specific preprocessing, which may be streamlined in future iterations.
Summary updated