Gene-Induced Multimodal Pre-training for Image-Omic Classification

Ting Jin, Xingran Xie, Renjie Wan, Qingli Li, Yan Wang

Abstract

Accurate classification of lung cancer subtypes is critical for personalized oncology. This paper presents GiMP (Gene-induced Multimodal Pre-training), a novel framework designed to integrate gigapixel histology images and high-dimensional genomic data. By leveraging efficient attention mechanisms and a gene-induced triplet learning strategy, GiMP achieves a state-of-the-art accuracy of 99.47% in distinguishing between Lung Adenocarcinoma (LUAD) and Lung Squamous Cell Carcinoma (LUSC).

Introduction

Modern oncological diagnosis relies on the synergy between histology (Whole Slide Images, WSIs) and genomics. However, integrating these modalities presents significant computational challenges. WSIs are "gigapixel" images containing tens of thousands of patches, leading to quadratic complexity in standard Transformers. Genomic data is high-dimensional, often comprising tens of thousands of genes, which risks overfitting. Finally, effectively fusing these disparate data types to capture deep biological relationships remains an open problem.

Methods

The GiMP framework addresses these challenges through three primary innovations:

  1. Group Multi-head Self Attention (GroupMSA): To process genomic data efficiently, genes are partitioned into non-overlapping fragments. Intra-group attention captures local patterns, while inter-group attention models global relationships.
  2. Masked Patch Modeling (MPM): The model is pre-trained by masking contiguous tissue patches and reconstructing their feature embeddings. This approach forces the model to learn biological context rather than superficial visual textures.
  3. Gene-induced Triplet Learning: This fusion strategy aligns summary tokens from both image and gene encoders. A triplet loss function ensures that samples from the same cancer category are clustered together in the feature space, while different categories are separated.

To maintain efficiency with large WSIs, the framework utilizes Nystrom-based attention, providing linear complexity for processing thousands of image patches.

Overview of the GiMP framework architecture, showing the integration of genomic and histology data through GroupMSA, Patch Aggregation, and Multimodal Fusion.

Results

The model was evaluated using The Cancer Genome Atlas (TCGA) dataset for Non-Small Cell Lung Cancer (NSCLC), comprising 946 image-omic pairs. GiMP achieved a benchmark-setting 99.47% accuracy, significantly outperforming existing multimodal methods such as MCAT (96.32%) and Pathomic Fusion (96.84%). Ablation studies confirmed that the GiMP architecture alone, even without pre-training, surpasses current state-of-the-art models.

t-SNE visualization of feature embeddings for LUAD and LUSC. GiMP (a) shows the most distinct class separation compared to variants without genomic data (b) and other self-supervised methods (c, d).

Visual analysis via t-SNE plots demonstrates that the inclusion of genomic data during pre-training acts as a powerful anchor, creating well-defined boundaries between cancer subtypes that are otherwise blurred in image-only models.

Discussion and Conclusion

GiMP demonstrates that high-level feature reconstruction and efficient attention mechanisms can successfully bridge the gap between histology and genomics. While the current model uses a fixed sequence length for pre-training and requires a supervised genomic pre-training step, its performance on the TCGA-NSCLC dataset is a significant step forward. Future work will focus on dynamic sequence lengths and fully end-to-end self-supervised learning across diverse cancer types.

Summary updated

BibTeX
@inproceedings{jin2023gene,
  author = {Ting Jin and Xingran Xie and Renjie Wan and Qingli Li and Yan Wang},
  title = {{Gene-Induced Multimodal Pre-training for Image-Omic Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43987-2\_49},
}
Figure