Abstract
Single-cell RNA sequencing (scRNA-seq) offers detailed insights into cellular heterogeneity. Recent advancements leverage single-cell large language models (scLLMs) for effective representation learning. These models focus exclusively on transcriptomic data, neglecting complementary biological knowledge from textual descriptions. To overcome this limitation, we propose scMMGPT, a novel multimodal framework designed for language-enhanced representation learning in single-cell transcriptomics. Unlike existing methods, scMMGPT employs robust cell representation extraction, preserving quantitative gene expression data, and introduces an innovative two-stage pre-training strategy combining discriminative precision with generative flexibility. Extensive experiments demonstrate that scMMGPT significantly outperforms unimodal and multimodal baselines across key downstream tasks, including cell annotation and clustering, and exhibits superior generalization in out-of-distribution scenarios.
Introduction
In computational biology, single-cell RNA sequencing (scRNA-seq) is a cornerstone technology used to measure gene expression levels at the individual cell level. This process generates massive, high-dimensional numerical matrices that characterize cellular states. While recent Single-Cell Large Language Models (scLLMs) have shown promise in learning representations from this data, they typically operate in a vacuum, ignoring the vast repository of biological knowledge encoded in natural language—such as tissue origins, disease states, and functional definitions found in scientific literature.
Existing attempts to bridge this gap often suffer from two critical flaws. First, they frequently discard precise quantitative gene expression values, converting them into simplified "cell sentences" (e.g., "Gene A is high"), which leads to significant information loss. Second, they often prioritize generative "chat" capabilities over the rigorous discriminative alignment required for precise scientific tasks like cell type annotation.
The scMMGPT Framework
To address these limitations, we introduce scMMGPT (Single-Cell MultiModal Generative Pre-trained Transformer). This framework is designed to align quantitative transcriptomics with semantic biological knowledge without sacrificing data fidelity.
Preserving Quantitative Precision
Rather than tokenizing gene names into text, scMMGPT utilizes a specialized pre-trained scLLM (scGPT) as its cell encoder. This allows the model to directly process raw, normalized gene expression values. By maintaining the numerical integrity of the data, scMMGPT captures subtle biological signals that are often lost in text-based conversions.
Two-Stage Pre-training Strategy
The model employs a novel two-stage training approach to ensure both precision and flexibility:
- Discriminative Stage: Utilizing contrastive learning (InfoNCE) and matching losses, the model learns to accurately pair a cell's RNA profile with its corresponding textual description among a set of negative samples.
- Generative Stage: The model is trained to translate between modalities, generating descriptive text from cell embeddings and synthesizing pseudo-expression profiles from textual prompts.
Key Insights and Results
A central finding of this research is that numerical data is fundamentally more informative than text-based "cell sentences" for raw input. Our analysis shows that converting expression profiles into text leads to a dramatic increase in error rates, particularly when attempting to distinguish between morphologically similar cell types.
Performance Benchmarks
- Cell Type Annotation: scMMGPT consistently outperformed both unimodal and multimodal baselines, achieving up to a 10% improvement in F1 scores on standard benchmarks like PBMC-3K.
- Zero-Shot Generalization: The model demonstrated remarkable robustness in out-of-distribution scenarios. On the Tabula Sapiens dataset, it achieved 49.1% Top-1 accuracy without any task-specific fine-tuning.
- Clustering and Batch Correction: In practical laboratory applications, scMMGPT successfully grouped cells by biological identity while effectively neutralizing technical batch effects, surpassing traditional tools like Seurat and scVI.
Conclusion
scMMGPT represents a significant step forward in single-cell analysis by successfully integrating textual biological knowledge with precise numerical transcriptomics. By preserving quantitative data and employing a dual-stage pre-training strategy, the model achieves state-of-the-art performance and robust generalization. Future work will focus on expanding the model to include non-human species, integrating multiomic data such as chromatin accessibility, and incorporating spatial context to further enrich cellular representations.
Summary updated