A Multimodal Graph Learning Framework for Versatile Spatial Transcriptomics Analysis with SpatialModal

Xingyi Li, Dongmin Zhao, Xingang Jia, Gaoming Du, Jialuo Xu, Qi Yang, Yiqi Chen, Yingfu Wu, Junnan Zhu, Wei Feng, Min Li, Xuequn ShangbioRxiv 2025 · DOI

TL;DR

SpatialModal is a novel multimodal graph learning framework that effectively integrates gene expression data with histological images to create unified spatial representations of tissues. This approach significantly improves the accuracy and interpretability of spatial transcriptomics analysis, enabling more precise dissection of complex biological structures and functions.

What Problem Does This Paper Address?

Spatial transcriptomics (ST) technologies allow us to study gene expression within its spatial context, offering unprecedented insights into tissue organization and cellular function. However, current methods often rely solely on gene expression data, neglecting the rich contextual information provided by histological images (like H&E stains). This limitation hinders the ability to distinguish between regions with similar gene expression profiles but different underlying morphologies, thus restricting the precise analysis of complex biological structures and disease microenvironments. The paper aims to bridge this gap by developing a framework that can effectively fuse gene expression and image data for more comprehensive and accurate spatial transcriptomics analysis.

Methods

SpatialModal is a multimodal graph learning framework built upon graph neural networks (GNNs). Here's a breakdown of its key components:

  • Hierarchical Encoding: It first uses separate GNNs to encode gene expression and image modalities independently, generating low-dimensional embeddings. These are then fed into a shared GCN to learn cross-modal joint representations. A graph variational autoencoder (VGAE) further integrates these representations to enhance expressiveness and spatial consistency.
  • Adaptive Cross-Modal Fusion: The framework employs an adaptive fusion mechanism to balance the weights and complementarity of expression and morphological information in the latent space.
  • Dual Contrastive Learning: To boost the discriminative power of the learned representations, SpatialModal uses a contrastive learning strategy. This involves creating augmented views of the data (by randomly dropping edges and features) and training the model to maximize consistency between positive pairs while minimizing it for negative pairs. It also incorporates a "topology-aware" contrastive learning approach that considers spatial adjacency.
  • Spatial Mapping Based Gene Denoising: A module inspired by STAGE is included to improve the biological consistency of denoised gene expression data by incorporating spatial mapping constraints.
  • Multi-Slice Integration: SpatialModal can integrate and train on multiple tissue slices simultaneously, forming a unified input for analysis.

Datasets

The authors evaluated SpatialModal on a variety of public ST datasets, showcasing its versatility across different tissue types, species, and resolutions:

  • Human DLPFC (10x Visium): Used for spatial domain identification and analysis of cortical laminar structures.
  • Human Breast Cancer (10x Visium): Utilized for heterogeneity analysis of the tumor microenvironment.
  • Mouse Brain (Anterior and Posterior, 10x Visium): Applied to analyze mouse brain structure and molecular resolution.
  • Mouse Somatosensory Cortex (osmFISH): Used for single-cell resolution ST data analysis.
  • Mouse Hypothalamic Preoptic Area (MERFISH): Another single-cell resolution dataset for evaluating performance.
  • Mouse Visual Cortex (STARmap): Employed to assess spatial transcriptomic analysis capabilities.
  • Mouse Embryo (Stereo-seq): Used for analyzing embryonic tissue development.

Key Insights and Ideas

  • Image Data is Crucial for Dissecting Complex Tissues: The paper strongly demonstrates that integrating histological image data alongside gene expression is essential for accurately distinguishing spatially distinct regions that might otherwise appear similar based on gene expression alone.
  • Hierarchical and Adaptive Fusion is Key: SpatialModal's success lies in its hierarchical encoding and adaptive fusion strategy, which allows it to effectively learn from and balance the contributions of different modalities.
  • Contrastive Learning Enhances Discriminative Power: The dual contrastive learning approach significantly improves the model's ability to generate robust and discriminative spatial representations.
  • Versatility Across Modalities and Resolutions: SpatialModal performs exceptionally well on datasets with and without histological images, highlighting its adaptability to various ST technologies and data types.
  • Uncovering Fine-Grained Biological Structures: The framework successfully resolves intricate structures like cortical layers, subiculum/post-subiculum, and specific nuclei, providing molecular-level insights.

Key Results

SpatialModal consistently outperformed seven state-of-the-art methods across multiple datasets and downstream tasks:

  • Spatial Domain Identification: Achieved superior clustering performance, with significantly higher Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI) scores on DLPFC (ARI=0.80 on slice 151672), breast cancer (ARI=0.64, NMI=0.72), and mouse brain datasets.
  • Gene Expression Reconstruction: Demonstrated high fidelity in reconstructing gene expression patterns, with Pearson correlation coefficients surpassing baselines.
  • Pseudotime Inference: Successfully elucidated developmental trajectories and hierarchical structures, such as the Layer 1 to Layer 2 to white matter trajectory in the DLPFC.
  • Biological Interpretation: Accurately identified and elucidated the metabolic-immune dynamic boundaries in breast cancer microenvironments and functional partitioning in the mouse cortex.
  • Robustness: Showed exceptional robustness in single-cell resolution analysis (osmFISH, MERFISH) and multi-slice integration tasks.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Image Data Dependency: Current image-based experiments are limited to 10x Visium datasets due to the requirement for precise pixel-level spatial coordinates. Future work aims to incorporate other ST platforms by exploring approximate image reconstruction or registration strategies.
  • Pretrained Image Encoder: The use of a general-purpose ResNet152 model might not fully capture domain-specific patterns in histological images. Future research could involve using domain-adaptive or self-supervised visual models trained directly on histological data.
  • Computational Complexity: The multimodal architecture leads to relatively long training and inference times, potentially limiting scalability. Future work will focus on model simplification techniques like knowledge distillation, sparse graph construction, or modular training to improve efficiency.

Summary updated

Abstract

ABSTRACT The development of spatial transcriptomics(ST) technologies enables the exploration of tissue structure and cellular function within a spatial context. However, mainstream analytical methods primarily focus on the gene expression modality itself and struggle to fully leverage auxiliary information from histological images, which limits precise dissection of complex biological structures. To address this, we propose SpatialModal, a multimodal graph learning framework that effectively integrates gene expression data and histological images from spatial transcriptomics to construct unified spatial representations. SpatialModal effectively captures synergistic interactions between modalities by jointly modeling intra- and inter-modal complementary features while incorporating spatial adjacency information. Furthermore, it employs a dual contrastive learning strategy to enhance the discriminative power of representations, thereby enabling efficient and robust analysis of tissue structures. We validate the effectiveness of this approach on multiple public datasets covering diverse tissue types, species, and resolutions. Experiments demonstrate that SpatialModal exhibits significant advantages in downstream tasks, including spatial domain identification, gene expression reconstruction, and pseudotime inference, and accurately elucidates the hierarchical structure of the mouse cortex as well as the metabolic-immune dynamic boundaries in the breast cancer microenvironment. Additionally, SpatialModal shows exceptional robustness in single-cell resolution and multi-slice integration tasks, providing an efficient and broadly applicable analytical tool for spatial transcriptomics research.
BibTeX
@misc{li2025multimodal,
  author = {Xingyi Li and Dongmin Zhao and Xingang Jia and Gaoming Du and Jialuo Xu and Qi Yang and Yiqi Chen and Yingfu Wu and Junnan Zhu and Wei Feng and Min Li and Xuequn Shang},
  title = {{A Multimodal Graph Learning Framework for Versatile Spatial Transcriptomics Analysis with SpatialModal}},
  year = {2025},
  doi = {10.1101/2025.05.11.653070},
  howpublished = {bioRxiv},
}
Figure