Cross-Modal Translation and Alignment for Survival Analysis

Fengtao Zhou, Hao ChenICCV 2023 · DOI · OpenReview

Introduction

Survival analysis—predicting the time until a clinical event like relapse or death—is a cornerstone of precision medicine. Historically, researchers have relied on either Whole Slide Images (WSIs), providing microscopic views of tumor tissue, or Genomic Profiles, offering molecular data such as RNA sequencing. While combining these modalities should theoretically provide a more complete clinical picture, current methods often fall short due to simple feature concatenation that ignores complex intrinsic correlations or restrictive guidance where genomics merely filters image features, potentially discarding vital pathological information.

The Cross-Modal Translation and Alignment (CMTA) framework addresses these challenges by effectively bridging the gap between pathological images and genomic data. By utilizing a novel encoder-decoder architecture that "translates" information between modalities and enforces feature alignment, the model captures deep biological correlations that simpler fusion methods miss.

Methods

The CMTA framework introduces a sophisticated "translation and alignment" pipeline designed to explore cross-modal correlations without losing the unique, complementary information each data source provides.

Overview of the CMTA framework architecture, showing the parallel pathology and genomics branches with cross-modal translation and alignment.

Parallel Encoder-Decoders

The model utilizes two separate branches for pathology and genomics. Each branch features an encoder to extract "intra-modal" features and a decoder to generate "cross-modal" representations. To handle the massive scale of gigapixel WSIs, the framework employs Nystrom attention to approximate global self-attention efficiently.

Cross-Modal Attention and Translation

The Cross-Modal Attention Module (CMAM) serves as the information bridge, using an attention mechanism to relate specific regions in a pathology image to specific gene groups. The decoders perform a "translation" task: the pathology decoder predicts genomic-like features from image data, and vice versa. This forces the model to learn the underlying biological mapping between the two modalities.

Alignment Constraints

To ensure translated features are accurate, the authors apply an $L_1$ distance loss to align translated features with actual features from the target modality. Crucially, a unidirectional optimization strategy (detaching the target from the gradient) is used to prevent the model from collapsing into a state where it only learns redundant, shared information.

Results

The model was evaluated on five public datasets from The Cancer Genome Atlas (TCGA): BLCA, BRCA, GBMLGG, LUAD, and UCEC. These datasets provide paired high-resolution tissue slides and molecular sequencing for hundreds of patients per cancer type.

Performance comparison (C-index) of CMTA against state-of-the-art unimodal and multimodal methods across five TCGA datasets.

CMTA consistently outperformed state-of-the-art (SOTA) methods, including MCAT and DualTrans, across all datasets using the C-index metric. Notably, on the UCEC dataset, CMTA achieved a C-index of 0.6975, representing a 6.39% improvement over the previous best multi-modal method. Unlike previous models that showed inconsistent performance across different cancer types, CMTA demonstrated stable, superior results across the board. Ablation studies further confirmed that removing the Cross-Modal Attention or Alignment Constraints led to significant performance drops, such as a 4.93% decrease on LUAD without the attention module.

Discussion and Conclusion

A key insight of this work is that translation acts as a regularizer. By forcing the image branch to "imagine" the genomic profile, the model learns robust features grounded in biological reality. Furthermore, the framework avoids the "mediocre modality trap" by allowing each modality to maintain its identity while sharing information through the attention bridge.

However, limitations remain. The "black box" nature of deep learning necessitates future work on interpretability to visualize which tissue structures map to specific gene clusters. Additionally, the model's requirement for both high-quality WSIs and genomic sequencing may limit its use in clinical settings where one modality is missing. Future research will focus on developing "missing modality" versions of the framework and further optimizing the computational intensity required for processing gigapixel images.

Summary updated

Abstract

With the rapid advances in high-throughput sequencing technologies, the focus of survival analysis has shifted from examining clinical indicators to incorporating genomic profiles with pathological images. However, existing methods either directly adopt a straightforward fusion of pathological features and genomic profiles for survival prediction, or take genomic profiles as guidance to integrate the features of pathological images. The former would overlook intrinsic cross-modal correlations. The latter would discard pathological information irrelevant to gene expression. To address these issues, we present a Cross-Modal Translation and Alignment (CMTA) framework to explore the intrinsic cross-modal correlations and transfer potential complementary information. Specifically, we construct two parallel encoder-decoder structures for multi-modal data to integrate intra-modal information and generate cross-modal representation. Taking the generated cross-modal representation to enhance and recalibrate intra-modal representation can significantly improve its discrimination for comprehensive survival analysis. To explore the intrinsic crossmodal correlations, we further design a cross-modal attention module as the information bridge between different modalities to perform cross-modal interactions and transfer complementary information. Our extensive experiments on five public TCGA datasets demonstrate that our proposed framework outperforms the state-of-the-art methods.
BibTeX
@inproceedings{zhou2023cross,
  author = {Fengtao Zhou and Hao Chen},
  title = {{Cross-Modal Translation and Alignment for Survival Analysis}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01964},
}
Figure