Abstract
A cell is governed by the interaction of myriads of macromolecules. Inferring such a network of interactions has remained an elusive milestone in cellular biology. Building on recent advances in large foundation models and their ability to learn without supervision, we present scPRINT, a large cell model for the inference of gene networks pre-trained on more than 50 million cells from the cellxgene database. Using innovative pretraining tasks and model architecture, scPRINT pushes large transformer models towards more interpretability and usability when uncovering the complex biology of the cell. Based on our atlas-level benchmarks, scPRINT demonstrates superior performance in gene network inference to the state of the art, as well as competitive zero-shot abilities in denoising, batch effect correction, and cell label prediction. On an atlas of benign prostatic hyperplasia, scPRINT highlights the profound connections between ion exchange, senescence, and chronic inflammation.
Introduction
Understanding the complex web of gene interactions—the Gene Regulatory Network (GRN)—is a cornerstone of systems biology. However, inferring these networks from single-cell RNA sequencing (scRNA-seq) data is fraught with challenges, including data sparsity (technical dropouts), high noise levels, and batch effects. While foundation models for single-cell data have recently emerged, they often lack the interpretability required for reliable network extraction or necessitate extensive task-specific fine-tuning. We present scPRINT (single-cell PRe-trained Inference of Networks with Transformers), a large-scale foundation model designed to overcome these hurdles by leveraging unprecedented scale and biological priors.
Methods: Architecture and Multi-Task Pre-training
scPRINT utilizes a bidirectional transformer architecture with several key innovations. First, it incorporates biological priors by representing genes using ESM2 protein embeddings. This ensures the model "knows" about the structural and evolutionary relationships between genes before processing expression data, significantly reducing trainable parameters while improving generalization.
The model is trained on over 54 million cells from the CELLxGENE database using three simultaneous pre-training tasks:
- Denoising as Upsampling: Learning to recover original expression profiles from downsampled, noisy versions.
- Bottleneck Learning: Reconstructing a cell's entire expression profile from a compressed "cell embedding" to learn the underlying grammar of cell states.
- Hierarchical Classification: Predicting metadata (cell type, disease, sex) to disentangle biological signals from technical noise.
Results: Benchmarking and Biological Discovery
scPRINT demonstrates state-of-the-art performance in gene network inference, outperforming existing models like scGPT and Geneformer v2. On average, it recovered 67% more connections than classical methods like GENIE3. A critical innovation is the "head selection" technique, which identifies biologically relevant attention heads within the transformer to improve network precision.
Beyond network inference, scPRINT exhibits powerful zero-shot capabilities. Without explicit training for classification, it achieved 62% accuracy in cell-type prediction on complex datasets, surpassing specialized tools like CellTypist. In a case study of human prostate tissue, scPRINT identified PAGE4 as a central hub and revealed network shifts in diseased fibroblasts, linking metal transport to cellular senescence and chronic inflammation. Despite its scale, the medium-sized model is efficient, requiring only a single A40 GPU for 48 hours of training.
Discussion
While scPRINT marks a significant advance, challenges remain in determining the exact directionality of gene regulation (e.g., inhibition vs. activation) from attention matrices alone. Furthermore, the scarcity of "perfect" ground-truth networks makes evaluation an ongoing challenge. Future work will focus on multi-omics integration, incorporating spatial transcriptomics and chromatin accessibility to provide a more holistic view of cellular regulation.
Summary updated