Abstract
Identifying the genetic and molecular drivers of phenotypic heterogeneity among individuals is vital for understanding human health and for diagnosing, monitoring, and treating diseases. To this end, international consortia such as the Human Cell Atlas and the Tabula Sapiens are creating comprehensive cellular references. Due to the massive volume of data generated, machine learning methods, especially transformer architectures, have been widely employed in related studies. However, applying machine learning to cellular data presents several challenges. One such challenge is making the methods interpretable with respect to both the input cellular information and its context. Another less explored challenge is the accurate representation of cells outside existing references, referred to as out-of-distribution (OOD) cells. The out-of-distribution could be attributed to various physiological conditions, such as comparing diseased cells, particularly tumor cells, with healthy reference data, or significant technical variations, such as using transfer learning from single-cell reference to spatial query data. Inspired by the global workspace theory in cognitive neuroscience, we introduce CellMemory, a bottlenecked Transformer with improved generalization capabilities designed for the hierarchical interpretation of OOD cells unseen during reference building. Even without pre-training, it exceeds the performance of large language models pre-trained with tens of millions of cells. In particular, when deciphering spatially resolved single-cell transcriptomics data, CellMemory demonstrates the ability to interpret data at the granule level accurately. Finally, we harness CellMemory’s robust representational capabilities to elucidate malignant cells and their founder cells in different patients, providing reliable characterizations of the cellular changes caused by the disease.
Introduction
The rapid expansion of single-cell sequencing has enabled the creation of massive reference atlases, such as the Human Cell Atlas and Tabula Sapiens. These resources provide a baseline for understanding healthy tissue, yet they present significant challenges when researchers attempt to map "out-of-distribution" (OOD) cells. OOD cells—those that deviate from the reference due to disease states (e.g., malignancy) or technical differences (e.g., spatial transcriptomics vs. dissociated single-cell data)—often fail to align correctly with standard models. Furthermore, the "black box" nature of many deep learning architectures makes it difficult for biologists to extract the specific gene regulatory programs driving these cellular states.
To address these hurdles, we present CellMemory, a bottlenecked Transformer architecture designed for high-resolution interpretation and robust generalization. By drawing inspiration from cognitive neuroscience, CellMemory moves beyond the computational inefficiencies of standard Transformers to provide a scalable, interpretable framework for single-cell analysis.
Methods: A Neuroscience-Inspired Architecture
CellMemory is built upon the Global Workspace Theory (GWT) from cognitive neuroscience. GWT posits that human consciousness arises from a "global workspace" where specialized modules compete to write information into a limited-capacity space, which is then broadcast back to the system.
The Bottleneck Mechanism
Traditional Transformers utilize self-attention mechanisms with quadratic complexity ($O(N^2)$), which becomes computationally prohibitive when processing thousands of genes (tokens) per cell. CellMemory replaces this with a cross-attention bottleneck. In this architecture:
- Specialist Modules: Individual genes act as specialists that "write" their information into a fixed number of Memory Slots ($H$).
- Write and Broadcast: Information is filtered through these slots and then "broadcast" back to the specialists.
- Efficiency: This iterative process reduces complexity to $O(M \cdot H)$, where $M$ is the number of genes and $H$ is the small number of memory slots.
This constraint forces the model to prioritize biologically significant signals over noise, effectively acting as a biological filter. Furthermore, CellMemory retains "zero-expression" data. While many models discard genes with zero counts to save memory, we found that these zeros provide critical context for integrating data across different sequencing platforms.
Hierarchical Interpretation and TAGs
One of the primary advantages of the bottlenecked approach is its inherent interpretability. The model generates "Memory Scores" for genes, allowing researchers to identify Top Attention Genes (TAGs). These TAGs represent the specific genetic drivers that define a cell's identity or its transition into a diseased state. We observed that different memory slots naturally specialize in distinct biological functions, such as lymphocyte activation or oxidative stress, providing a hierarchical view of cellular activity.
Experimental Results
We validated CellMemory using a massive dataset of over 15 million cells across various species and conditions, including the SEA-AD (Alzheimer’s) and AIDA (Asian Immune Diversity Atlas) datasets.
Superior Performance over LLMs
Despite not requiring the extensive pre-training typical of large language models (LLMs) like scGPT or Geneformer, CellMemory achieved higher accuracy in identifying rare cell types. In the SEA-AD dataset, which contains 131 distinct cell types, CellMemory reached approximately 90% accuracy, significantly outperforming competing tools like scPoli, which achieved roughly 60%.
Rare Cell and Spatial Discovery
CellMemory demonstrated remarkable sensitivity in the AIDA dataset by identifying ASDC (Axl-positive signature dendritic cells), a rare immune cell type constituting only 0.1% of the population. This population had been missed by previous manual annotations. In spatial omics applications, the model provided precise TAGs that validated known markers of tumor invasion, such as KRT15, at the boundaries of breast cancer tissues.
Tracing Malignancy
By leveraging its robust representational capabilities, CellMemory successfully elucidated the transition from healthy alveolar (AT2) cells to malignant tumor cells in lung cancer patients. This ability to trace "founder cells" and intermediate states provides a reliable characterization of the cellular changes driven by disease progression.
Discussion and Future Work
CellMemory demonstrates that computational constraints, when inspired by biological principles, can lead to more efficient and interpretable machine learning models. By focusing on a limited "memory space," the model avoids the overfitting and noise sensitivity common in larger architectures.
While currently a supervised framework, future iterations of CellMemory will explore self-supervised learning to further reduce pre-training costs. Additionally, we aim to extend the architecture to multimodal data, integrating transcriptomics with proteomics and epigenomics (ATAC-seq) to provide a truly holistic view of the cellular landscape.
Summary updated