scMBERT: A Pre-Trained Deep Learning Model for Single-Cell Multiomic Data Representation and Prediction (Student Abstract)

Xiaojian Chen, Kuai Yu, Min-Zhi Jiang, Cihan Xiao, Ziqi Fu, Weiqiang ZhouAAAI 2025 · DOI

Abstract

Single-cell multiomic data, combining gene expression (scRNA-seq) and chromatin accessibility (scATAC-seq), provides a comprehensive view of cellular states and regulatory logic. However, integrating these high-dimensional, sparse, and noisy modalities remains a challenge. We present scMBERT, a BERT-based framework pre-trained on multiomic data using a novel modality prediction task. scMBERT achieves state-of-the-art performance in batch-effect correction and competitive accuracy in cell type annotation, demonstrating the power of cross-modal consistency in single-cell analysis.

Introduction

Single-cell RNA sequencing (scRNA-seq) has transformed our understanding of cellular diversity by profiling gene expression. Yet, RNA alone only captures the output of cellular processes. To understand the underlying regulatory mechanisms, researchers increasingly turn to multiomic assays like scATAC-seq, which measures the "regulome" via chromatin accessibility.

The integration of these modalities is difficult due to their extreme sparsity and high dimensionality. While Large Language Models (LLMs) like BERT and GPT have been adapted for single-cell data (e.g., scBERT, scGPT), most focus primarily on RNA or are constrained by fixed gene sets. scMBERT addresses these limitations by providing a flexible transformer architecture capable of integrating multiple cellular "views" simultaneously.

Methods

The scMBERT framework adapts the Bidirectional Encoder Representations from Transformers (BERT) architecture for multiomic sequences. Unlike models that process fixed gene sets, scMBERT handles variable inputs by processing only non-zero features, enhancing computational efficiency and cross-dataset flexibility.

The scMBERT architecture includes a self-supervised pre-training phase with Masked Value Prediction (MVP) and Data Modality Prediction (DMP) tasks, followed by supervised fine-tuning for specific downstream tasks.

The training process consists of two stages:

  1. Self-supervised Pre-training: The model learns biological representations from unlabeled data through two tasks:
  2. Masked Value Prediction (MVP): A standard BERT-style task where the model predicts masked expression or accessibility values.
  3. Data Modality Prediction (DMP): A novel task requiring the model to identify the source (RNA or ATAC) of a data segment. This serves as a regularizer to ensure cross-modal consistency.
  4. Supervised Fine-tuning: The pre-trained encoder is coupled with a classification head for tasks such as cell type annotation.

Experimental Setup

The model was pre-trained on the ENCODE 4 portal, encompassing 47 single-cell multiomic datasets across 13 tissue types. Evaluation was conducted using the NeurIPS BMMC (Bone Marrow Mononuclear Cells) dataset, containing approximately 69,000 cells from 12 donors. To assess generalizability, data from three donors were held out during training.

Results

scMBERT was benchmarked against leading single-cell models, scBERT and scGPT, focusing on cell type annotation accuracy, biological signal preservation (AvgBIO), and batch effect correction (AvgBATCH).

Performance comparison showing scMBERT's superior batch correction and biological signal preservation compared to existing single-omic models.

Key findings include:

  • Accuracy: scMBERT (Multiome) achieved 75.56% accuracy, comparable to the 76.68% achieved by scGPT, a significantly larger generative model.
  • Batch Correction: scMBERT significantly outperformed competitors with an AvgBATCH score of 0.9072 (vs. 0.8039 for scGPT and 0.8696 for scBERT). This indicates a superior ability to merge data across different donors and technical batches.
  • Biological Representation: With an AvgBIO score of 0.6362, scMBERT demonstrated better preservation of distinct biological signals compared to scGPT (0.5272).

Discussion

The results underscore the advantages of multiomic integration. By incorporating both the transcriptome and the regulome, scMBERT gains a more robust understanding of cellular identity. The introduction of the DMP task effectively prevents overfitting to a single modality.

Future work will focus on scaling pre-training to larger multiomic datasets and extending the model's capabilities to tasks such as missing data imputation (e.g., predicting ATAC-seq from RNA-seq) and signal enhancement. Currently, the requirement for "unified open chromatin regions" necessitates specific preprocessing, which may be streamlined in future iterations.

Summary updated

BibTeX
@inproceedings{chen2025scmbert,
  author = {Xiaojian Chen and Kuai Yu and Min-Zhi Jiang and Cihan Xiao and Ziqi Fu and Weiqiang Zhou},
  title = {{scMBERT: A Pre-Trained Deep Learning Model for Single-Cell Multiomic Data Representation and Prediction (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35242},
}
Figure