Cell segmentation-free inference of cell types from in situ transcriptomics data

Jeongbin Park, Wonyl Choi, Sebastian Tiesmeyer, Brian Long, Lars E. Borm, Emma Garren, Thuc Nghi Nguyen, Bosiljka Tasic, Simone Codeluppi, Tobias Graf, Matthias Schlesner, Oliver Stegle, Roland Eils, Naveed IshaquebioRxiv 2019 · DOI

TL;DR

This paper introduces SSAM, a novel computational framework that infers cell types and tissue domains directly from spatial transcriptomics data without requiring cell segmentation. SSAM overcomes limitations of existing methods by directly analyzing mRNA signals, leading to more accurate and comprehensive cell type identification, especially for challenging regions and rare cell types.

What Problem Does This Paper Address?

Spatial transcriptomics technologies allow us to map gene expression within intact tissues, providing a bridge between cellular identity and spatial organization. A key step in analyzing this data is identifying different cell types and their locations. Traditional approaches rely on cell segmentation – essentially drawing boundaries around individual cells – to assign gene expression profiles. However, this process is prone to errors, especially with densely packed cells, uneven borders, or complex tissue structures. Inaccurate segmentation can lead to missed cells, incorrect cell-type assignments, and incomplete tissue maps. This paper addresses this challenge by developing a method that bypasses the need for segmentation altogether, directly inferring cell types from the spatial distribution of mRNA molecules.

Methods

The core of the paper is the Spot-based Spatial cell-type Analysis by Multidimensional mRNA density estimation (SSAM) framework. SSAM operates in four main steps:

  1. mRNA Signal Estimation: SSAM first estimates the spatial density of mRNA for each gene across the tissue using Kernel Density Estimation (KDE) with a Gaussian kernel. This creates a "gene expression vector field," where each point (pixel/voxel) in the tissue is represented by a vector encoding the estimated density of multiple genes.
  2. Cell-Type Signature Identification: SSAM identifies cell-type signatures by clustering these gene expression vectors. This can be done de novo (discovering new signatures from the data) or by using pre-defined signatures from other sources (like scRNA-seq or previous segmentation-based methods). A key aspect here is downsampling the gene expression vectors to focus on local maxima, improving computational efficiency.
  3. Cell-Type Map Generation: The identified cell-type signatures are then mapped back onto the gene expression vector field. Each pixel/voxel in the tissue is assigned a cell type based on the correlation between its gene expression profile and the known cell-type signatures. This creates a "cell-type map."
  4. Tissue Domain Construction: Finally, SSAM identifies tissue domains by analyzing the spatial composition of cell types within sliding windows across the cell-type map. Clustering these domain compositions reveals distinct tissue regions (e.g., cortical layers).

The novelty lies in the segmentation-free approach, the use of KDE for spatial density estimation, and the direct mapping of gene expression signals to cell types and tissue domains.

Datasets

The authors applied SSAM to three publicly available mouse brain datasets:

  • Somatosensory cortex (SSp) from the osmFISH dataset.
  • Hypothalamic preoptic region (POA) from the MERFISH dataset.
  • Visual cortex (VISp) from a newly generated multiplexed smFISH dataset.

These datasets represent different spatial transcriptomics techniques and dimensionality (2D and 3D), allowing SSAM's robustness to be tested across various scenarios.

Key Insights and Ideas

  • Segmentation is a Bottleneck: The paper strongly argues that cell segmentation is a major limitation in current spatial transcriptomics analysis, leading to missed cells and inaccurate cell-type calls, especially in complex tissues.
  • Direct Signal Analysis is Powerful: SSAM demonstrates that directly analyzing the spatial distribution of mRNA signals, without relying on cell boundaries, can yield more accurate and comprehensive results.
  • Improved Detection of Rare Cells and Structures: SSAM showed a remarkable ability to detect cell types and structures that were missed or poorly resolved by previous segmentation-based methods, such as astrocytes in the SSp and specific neuronal subtypes in the VISp.
  • Unified Framework for 2D and 3D: SSAM's ability to handle both 2D and 3D data with consistent parameters highlights its flexibility and broad applicability across different spatial transcriptomics modalities.
  • De Novo Discovery: The de novo mode of SSAM allows for the discovery of novel cell types and subtypes that might not be present in existing reference datasets.

Key Results

  • Improved Cell Type Mapping: SSAM achieved high matching scores when compared to segmentation-based methods, indicating strong agreement, but also identified cell types and regions missed by those methods. For example, SSAM detected significantly more Mfge8-expressing astrocytes in the SSp than previously reported.
  • Accurate Domain Identification: SSAM successfully identified known cortical layers in the SSp and distinct neuronal clusters within the VISp, demonstrating its capability in defining tissue architecture.
  • Robustness Across Datasets: SSAM performed consistently across osmFISH (2D), MERFISH (3D), and smFISH datasets, often using similar parameters, showcasing its generalizability.
  • Identification of Rare Cell Types: SSAM successfully identified rare cell types like Sst Chodl in the VISp, validating its sensitivity for low-abundance populations.

Limitations and Future Work

  • Distinguishing Cell Growth vs. Proliferation: The authors acknowledge that for specific biological questions, such as distinguishing cell growth from proliferation or identifying multinucleation, SSAM might need to be used in conjunction with segmentation-based methods. They suggest SSAM's output can be used to validate segmentation or refine it.
  • Limited Gene Panels: For datasets with a limited number of profiled genes (e.g., multiplexed smFISH), SSAM might struggle to correctly assign some cell types due to insufficient marker gene information, potentially leading to incorrect correlations.
  • Future Work: The authors suggest SSAM can be a complementary tool for segmentation-based analysis and can be extended with existing pipelines. The framework's flexibility and robustness open doors for analyzing a wider range of spatial transcriptomics data and addressing more complex biological questions.

Summary updated

Abstract

Abstract Multiplexed fluorescence in situ hybridization techniques have enabled cell-type identification, linking transcriptional heterogeneity with spatial heterogeneity of cells. However, inaccurate cell segmentation reduces the efficacy of cell-type identification and tissue characterization. Here, we present a novel method called Spot-based Spatial cell-type Analysis by Multidimensional mRNA density estimation (SSAM), a robust cell segmentation-free computational framework for identifying cell-types and tissue domains in 2D and 3D. SSAM is applicable to a variety of in situ transcriptomics techniques and capable of integrating prior knowledge of cell types. We apply SSAM to three mouse brain tissue images: the somatosensory cortex imaged by osmFISH, the hypothalamic preoptic region by MERFISH, and the visual cortex by multiplexed smFISH. We found that SSAM detects regions occupied by known cell types that were previously missed and discovers new cell types.
BibTeX
@misc{park2019cell,
  author = {Jeongbin Park and Wonyl Choi and Sebastian Tiesmeyer and Brian Long and Lars E. Borm and Emma Garren and Thuc Nghi Nguyen and Bosiljka Tasic and Simone Codeluppi and Tobias Graf and Matthias Schlesner and Oliver Stegle and Roland Eils and Naveed Ishaque},
  title = {{Cell segmentation-free inference of cell types from <i>in situ</i> transcriptomics data}},
  year = {2019},
  doi = {10.1101/800748},
  howpublished = {bioRxiv},
}
Figure