A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction

Xiaofeng Liu, Kaiwen Tan, Weixian Huang, Xiaofeng Liu 0014, Jinlong Hu, Shoubin DongArtif. Intell. Medicine 2022 · DOI

TL;DR

MultiCoFusion is a new deep learning framework that predicts cancer survival and tumor grade by integrating histopathological images with genomic data. By using a multi-task learning strategy that alternates between these two related objectives, the model captures hidden correlations between morphology and molecular profiles, significantly outperforming traditional single-task and single-modality methods.

What Problem Does This Paper Address?

Predicting how a cancer patient will fare (prognosis) is notoriously difficult because tumors are heterogeneous. Pathologists traditionally rely on histopathological images (visualizing cell shapes and tissue patterns), while researchers increasingly use genomic data (like mRNA expression) to understand the molecular drivers of the disease.

The authors identify two major gaps in current research:

  1. Modality Isolation: Many models use either images or genes, but rarely both effectively. Even when they do, they often fail to capture the complex interactions between genes.
  2. Task Isolation: Survival analysis (predicting time-to-event) and cancer grading (classifying malignancy levels) are usually treated as independent problems. However, these tasks are biologically linked—a higher-grade tumor typically leads to shorter survival. Ignoring this correlation wastes valuable information.

Methods

The authors propose MultiCoFusion, which uses a "divide and conquer" approach for feature extraction followed by a shared fusion layer.

  • Image Branch: They utilize a pre-trained ResNet-152 to extract deep features from 1,024×1,024 pixel regions of interest (ROIs) from whole-slide images. They chose ResNet-152 over the more common VGG-19 because it is deeper yet more computationally efficient.
  • Genomic Branch: Instead of a standard "black box" neural network, they use a Sparse Graph Convolutional Network (SGCN). This network incorporates a biological prior: a gene-gene interaction graph from the HINT database. This ensures the model learns representations based on how genes actually interact in nature, rather than treating them as an unstructured list.
  • Fusion and Multi-task Sharing: The features from both branches are concatenated and fed into a shared Fully Connected Neural Network (FCNN).
  • Alternate Training: This is a key innovation. Instead of optimizing a single combined loss function, the model uses an alternate training scheme. In one iteration, it optimizes for survival (using Cox partial likelihood); in the next, it optimizes for cancer grade (using negative log-likelihood). This prevents one task from dominating the learning process and helps the model find a robust, shared representation.

Datasets

The framework was evaluated using glioma (brain tumor) datasets from The Cancer Genome Atlas (TCGA):

  • TCGA-LGG (Lower-Grade Glioma) and TCGA-GBM (Glioblastoma).
  • The final dataset included 953 samples from 469 patients, featuring both whole-slide images and mRNA expression data.
  • HINT Database: Used to construct the gene-gene interaction graph (containing over 62,000 protein-protein interactions) for the SGCN.

Key Insights and Ideas

  • The "Aha" of Multi-tasking: The authors found that multi-task learning acts as a powerful regularizer. Even when using only a single data modality (just images or just genes), the model performed better when trained on both survival and grading simultaneously than when trained on either task alone.
  • Biological Priors Matter: The SGCN outperformed standard dense networks. By forcing the network to respect known gene interactions, the model avoids overfitting to the high-dimensional noise inherent in genomic data.
  • Granularity Synergy: Survival analysis is a "fine-grained" regression task, while grading is a "coarse-grained" classification task. The authors suggest that the shared network learns to extract features that are robust enough to satisfy both levels of granularity.

Key Results

MultiCoFusion achieved state-of-the-art results on the glioma dataset:

  • Survival Prediction: Achieved a C-index of 0.857, outperforming the previous benchmark, Pathomic Fusion (0.832).
  • Cancer Grading: Achieved a micro-AUC of 0.923, a significant jump over single-task multi-modal methods (which hovered around 0.85).
  • F1 Score for Grade IV: The model was exceptionally accurate at identifying the most severe cases (Grade IV), reaching an F1 score of 0.998.
  • Qualitative Success: Kaplan-Meier survival curves showed that MultiCoFusion could clearly separate patients into low, medium, and high-risk groups more effectively than ground-truth histological grades alone.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • The "See-Saw" Effect: In multi-task learning, there is often a trade-off where improving one task might slightly hinder another. While MultiCoFusion improved both, finding the optimal balance as more tasks are added remains a challenge.
  • Generalization: The study focused exclusively on gliomas. Future work is needed to determine if this specific architecture (especially the SGCN component) generalizes to other cancer types like breast or lung cancer.
  • Interpretability: While the SGCN uses a graph, the final fusion layer is still a dense neural network. Making the "fusion" step more interpretable (e.g., identifying which specific image features correlate with which gene clusters) would be a major step forward for clinical adoption.

Summary updated

BibTeX
@article{liu2022multi,
  author = {Xiaofeng Liu and Kaiwen Tan and Weixian Huang and Xiaofeng Liu and Jinlong Hu and Shoubin Dong},
  title = {{A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction}},
  journal = {Artif. Intell. Medicine},
  year = {2022},
  doi = {10.1016/j.artmed.2022.102260},
}
Figure