Image-Enhanced Hybrid Encoding with Reinforced Contrastive Learning for Spatial Domain Identification in Spatial Transcriptomics

Daoyuan Wang, Lu Gao, Wenlan Chen, Cheng Liang, Fei GuoIJCAI 2025 · DOI

Introduction

Spatial transcriptomics (ST) has revolutionized our understanding of tissue architecture by mapping gene expression within its physical context. A critical task in ST analysis is the identification of "spatial domains"—functionally distinct regions such as brain layers or tumor microenvironments. However, existing computational methods often struggle to effectively integrate the multimodal nature of ST data, which includes high-dimensional gene expression, spatial coordinates, and high-resolution histological images. Many current approaches either ignore image data or use simplistic fusion techniques that fail to capture the complex interdependencies between a cell's genetic profile and its morphology. Furthermore, standard contrastive learning techniques often mistakenly penalize biologically similar cells as "negatives," degrading model performance.

Methodology

To address these challenges, we present IE-HERCL (Image-Enhanced Hybrid Encoding with Reinforced Contrastive Learning). The framework utilizes a sophisticated multi-stage pipeline designed for robust multimodal integration:

  • Hybrid Dual-Encoding: The model employs Autoencoders (AE) to capture non-spatial features and GraphSAGE encoders to model spatial dependencies. This dual-encoding is applied independently to both gene expression data and image features (pre-extracted via ResNet50).
  • Cross-Modal Triple-Attention: Rather than simple concatenation, IE-HERCL uses a triple-attention mechanism to adaptively fuse gene and image representations, ensuring the final latent embedding is coherent and biologically informative.
  • Reinforced Contrastive Learning: To solve the "false negative" problem, the authors implement high-order random walks on an affinity graph. This allows the model to identify and group biologically similar samples even if they are not identical views, preventing the model from being penalized for grouping similar cells together.
  • Optimal Transport (OT) Refinement: The model uses Sinkhorn divergence to dynamically align the learned clustering distribution with an auxiliary distribution, effectively "sharpening" the boundaries of identified spatial domains.
The IE-HERCL framework architecture, illustrating the hybrid encoding of gene expression and image data, followed by cross-modal attention and reinforced contrastive learning.

Experimental Results

IE-HERCL was evaluated across five diverse spatial transcriptomics datasets, including NanoString Lung 9-1 and various brain and cancer tissues. The model achieved state-of-the-art (SOTA) performance across all benchmarks, measured by the Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI).

Quantitative comparison of IE-HERCL against state-of-the-art methods across four major spatial transcriptomics datasets, demonstrating superior performance in ARI and other clustering metrics.

Key findings from the experimental evaluation include:

  • Superior Accuracy: On the Human Breast Cancer dataset, IE-HERCL outperformed the second-best method (stDCL) by 4.66% in ARI.
  • Robustness: The model demonstrated consistent performance in complex tissue structures where traditional methods like Scanpy or STAGATE produced fragmented or noisy results.
  • Ablation Success: Studies confirmed that the inclusion of image data and the reinforced contrastive learning strategy are essential components, as removing either significantly degraded the quality of domain identification.

Conclusion

IE-HERCL represents a significant advancement in spatial domain identification by effectively leveraging histological images and accounting for biological similarity within a contrastive learning framework. While the computational complexity of random walks and Optimal Transport is a factor for very large datasets, the model's ability to produce clean, biologically plausible tissue maps makes it a powerful tool for spatial biology. Future work may involve integrating additional modalities like spatial proteomics and utilizing domain-specific pre-trained image models to further enhance performance.

Summary updated

BibTeX
@inproceedings{wang2025image,
  author = {Daoyuan Wang and Lu Gao and Wenlan Chen and Cheng Liang and Fei Guo},
  title = {{Image-Enhanced Hybrid Encoding with Reinforced Contrastive Learning for Spatial Domain Identification in Spatial Transcriptomics}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/864},
}
Figure