Abstract
Multiplexed fluorescence in situ hybridization techniques have enabled cell-type identification, linking transcriptional heterogeneity with spatial heterogeneity of cells. However, inaccurate cell segmentation reduces the efficacy of cell-type identification and tissue characterization. Here, we present a method called Spot-based Spatial cell-type Analysis by Multidimensional mRNA density estimation (SSAM), a robust cell segmentation-free computational framework for identifying cell-types and tissue domains in 2D and 3D. SSAM is applicable to a variety of in situ transcriptomics techniques and capable of integrating prior knowledge of cell types. We apply SSAM to three mouse brain tissue images: the somatosensory cortex imaged by osmFISH, the hypothalamic preoptic region by MERFISH, and the visual cortex by multiplexed smFISH. Here, we show that SSAM detects regions occupied by known cell types that were previously missed and discovers new cell types.
Introduction: The Segmentation Bottleneck
In the burgeoning field of spatial transcriptomics (e.g., MERFISH, osmFISH), researchers aim to map exactly which genes are active in specific locations within a tissue. Traditionally, this requires cell segmentation: drawing digital boundaries around individual cells to assign mRNA "spots" to a specific nucleus or cytoplasm.
However, segmentation is a major bottleneck. Cells in complex tissues like the brain are often tightly packed, overlapping, or vary wildly in shape and signal intensity. If the segmentation algorithm fails to draw a perfect line, meaningful data is discarded, and rare or low-signal cell types (like astrocytes) are often missed entirely. The authors address this by asking: Can we identify cell types and tissue structures using the spatial distribution of mRNA alone, skipping the segmentation step?
The SSAM Framework
SSAM (Spot-based Spatial cell-type Analysis by Multidimensional mRNA density estimation) moves away from discrete cell boundaries toward a continuous "gene expression vector field."
The workflow consists of four main steps:
- mRNA Density Estimation: Using Kernel Density Estimation (KDE) with a Gaussian kernel, the discrete locations of mRNA molecules for each gene are converted into a continuous density map. These maps are stacked to create a multichannel "vector field" where every pixel encodes an expected gene expression profile.
- Signature Identification: SSAM identifies cell-type signatures either de novo (by clustering local maxima in the vector field using algorithms like Louvain or DBSCAN) or in a "guided mode" using prior scRNA-seq data.
- Cell-Type Mapping: Every pixel in the image is classified into a cell type based on its correlation with the identified signatures, creating a high-resolution "cell-type map."
- Domain Definition: A sliding window analyzes the composition of cell types across the map to identify larger tissue structures, such as the distinct layers of the cerebral cortex.
Validation and Datasets
The framework was validated across three distinct mouse brain datasets:
- osmFISH: 2D data of the somatosensory cortex (SSp) targeting 33 marker genes.
- MERFISH: 3D data of the hypothalamic preoptic region (POA) targeting 135 genes.
- Multiplexed smFISH: A newly generated dataset of the visual cortex (VISp) targeting 22 genes, used to demonstrate SSAM’s ability to discover novel sub-structures.
Key Results and Biological Insights
SSAM achieved high "matching scores" (0.67 to 0.88) when compared to traditional segmentation-based maps, proving it is a reliable alternative. Crucially, it excels in regions where traditional methods struggle.
- Recovering "Invisible" Cells: Traditional segmentation often misses astrocytes because they have lower mRNA density compared to neurons. SSAM successfully identified a significantly higher proportion of astrocytes (7–14% of tissue layers) by looking at the total density field rather than individual cell "blobs."
- Superior Ventricle Mapping: In the brain's ventricle regions, where cells are extremely small and tightly packed, SSAM correctly identified ependymal and choroid plexus cells that the original watershed segmentation algorithms missed.
- Rare Cell Detection: The algorithm successfully localized the rare Sst Chodl cell type to the L6-1 layer of the visual cortex, matching known biological literature.
- Discovery of Sub-layers: In the visual cortex, SSAM identified two distinct sub-types of Layer 4 (L4) neurons—Superficial and Deep—that were previously grouped together, providing a more nuanced view of cortical organization.
Conclusion and Future Directions
While SSAM is powerful for mapping where cell types are, it is not a replacement for segmentation in all contexts. For example, if a researcher needs to count the exact number of cells or study cell-specific traits like multinucleation, segmentation is still necessary.
However, SSAM can be used as a complementary tool: first, use SSAM to generate a "ground truth" of cell-type distributions, then use that information to refine and validate segmentation masks. Its 3D scalability and efficiency make it a robust, segmentation-free analysis option for the next generation of spatial transcriptomics experiments.
Summary updated