Abstract
Accurate classification of lung cancer subtypes is critical for personalized oncology. This paper presents GiMP (Gene-induced Multimodal Pre-training), a novel framework designed to integrate gigapixel histology images and high-dimensional genomic data. By leveraging efficient attention mechanisms and a gene-induced triplet learning strategy, GiMP achieves a state-of-the-art accuracy of 99.47% in distinguishing between Lung Adenocarcinoma (LUAD) and Lung Squamous Cell Carcinoma (LUSC).
Introduction
Modern oncological diagnosis relies on the synergy between histology (Whole Slide Images, WSIs) and genomics. However, integrating these modalities presents significant computational challenges. WSIs are "gigapixel" images containing tens of thousands of patches, leading to quadratic complexity in standard Transformers. Genomic data is high-dimensional, often comprising tens of thousands of genes, which risks overfitting. Finally, effectively fusing these disparate data types to capture deep biological relationships remains an open problem.
Methods
The GiMP framework addresses these challenges through three primary innovations:
- Group Multi-head Self Attention (GroupMSA): To process genomic data efficiently, genes are partitioned into non-overlapping fragments. Intra-group attention captures local patterns, while inter-group attention models global relationships.
- Masked Patch Modeling (MPM): The model is pre-trained by masking contiguous tissue patches and reconstructing their feature embeddings. This approach forces the model to learn biological context rather than superficial visual textures.
- Gene-induced Triplet Learning: This fusion strategy aligns summary tokens from both image and gene encoders. A triplet loss function ensures that samples from the same cancer category are clustered together in the feature space, while different categories are separated.
To maintain efficiency with large WSIs, the framework utilizes Nystrom-based attention, providing linear complexity for processing thousands of image patches.
Results
The model was evaluated using The Cancer Genome Atlas (TCGA) dataset for Non-Small Cell Lung Cancer (NSCLC), comprising 946 image-omic pairs. GiMP achieved a benchmark-setting 99.47% accuracy, significantly outperforming existing multimodal methods such as MCAT (96.32%) and Pathomic Fusion (96.84%). Ablation studies confirmed that the GiMP architecture alone, even without pre-training, surpasses current state-of-the-art models.
Visual analysis via t-SNE plots demonstrates that the inclusion of genomic data during pre-training acts as a powerful anchor, creating well-defined boundaries between cancer subtypes that are otherwise blurred in image-only models.
Discussion and Conclusion
GiMP demonstrates that high-level feature reconstruction and efficient attention mechanisms can successfully bridge the gap between histology and genomics. While the current model uses a fixed sequence length for pre-training and requires a supervised genomic pre-training step, its performance on the TCGA-NSCLC dataset is a significant step forward. Future work will focus on dynamic sequence lengths and fully end-to-end self-supervised learning across diverse cancer types.
Summary updated