Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation

Zhaorui Tan, Xi Yang, Tan Pan, Tianyi Liu, Chen Jiang, Xin Guo, Qiufeng Wang, Anh Nguyen, Yuan Qi, Kaizhu Huang, Yuan Cheng

TL;DR

This paper introduces PUIR (Personalized Universal Invariant Representation), a novel two-stage framework designed to tackle the challenge of cross-modality generalization in 3D medical imaging. By moving away from population-average models and instead learning an individual-level, personalized invariant representation, PUIR achieves state-of-the-art performance in both modality transfer and missing-modality segmentation across highly diverse imaging types.

What Problem Does This Paper Address?

In medical AI, a model trained on one type of scan (modality) often struggles to generalize to another. This difficulty is compounded by two distinct layers of variation:

  1. Modality Variations: The difference between structural scans (like MRI sequences or CT) and functional biochemical scans (like PET).
  2. Individual Variations: The unique anatomical and metabolic differences between patients that deviate from population averages.

The authors categorize generalization into two types: homogeneous (within similar structural sequences, like different MRI modalities) and heterogeneous (across entirely different structural and functional modalities, like CT to PET).

Existing methods typically focus on class-level or population-wide anatomical invariance, completely ignoring individual differences. This paper addresses this gap by asking: Can we learn a personalized representation that captures an individual's unique biological profile across all modalities?

Figure 1. A diagram of medical modalities and individual differ- ences. These variations are significant and deserve further study by the medical intelligence community.

Methods

The authors propose a two-stage approach called PUIR:

1. Pre-training Stage (Personalization)

Instead of forcing the network to learn a single modality-agnostic feature vector (which often erases crucial modality-specific details), PUIR aims to approximate a personalized invariant representation ($\mathbb{X}_h$).

  • Learnable Biological Prior ($\mathbb{O}$): An initialized tensor acts as a global biological dictionary. Encoders extract features from an available modality, which then use an attention mechanism to retrieve "missing" knowledge from $\mathbb{O}$ to complete the patient's full profile.
  • Three Key Constraints: To ensure $\mathbb{X}_h$ is robust, the framework applies:
  • Invariance Constraint: Ensures representations of the same individual from different modalities align closely.
  • Equivariance Constraint: Forces the model to mathematically preserve geometric transformations (like patient rotation or posture changes).
  • Decomposition Constraint: Ensures that the learned invariant representation retains enough detail to be successfully reconstructed back into any specific target modality.

2. Fine-tuning Stage (Downstream Adaptation)

The pre-trained modality encoders are then adapted to standard downstream tasks (such as segmentation or image synthesis) using standard task losses combined with the learned invariance loss to bridge the gap between seen and unseen patients.

Figure 2. A diagram of patient domains and medical modalities linked to our method. In the Pre-training stage, static mappings between domains and modalities are implicitly learned. The fine- tuning stage addresses gaps between seen and unseen patients.

Datasets

The framework was rigorously evaluated on several multi-modal 3D datasets:

  • BraTS23 & BraTS18: Brain tumor segmentation datasets featuring four structural MRI modalities (T1, T1ce, T2, FLAIR).
  • AutoPET-II: A dataset containing paired structural CT and functional AC-PET scans.
  • HNSCC & NSCLC (from TCIA): Head and neck / lung cancer datasets used to test a highly complex scenario involving non-attenuation-corrected PET (NAC-PET), attenuation-corrected PET (AC-PET), and CT scans.

Key Insights and Ideas

  • The Flaw of Modality-Agnostic Features: Previous methods tried to compress features into a space ($Z_h$) completely independent of the modality. The authors point out that this completely erases modality-specific details, making it impossible to reconstruct the original images accurately. PUIR's decomposition constraint solves this.
  • Static Biological Mappings: The core philosophical insight is that while individuals differ wildly from each other, the underlying biological mapping from a single individual's profile to any medical modality remains static.
  • Equivariance vs. Invariance Conflict: The authors discovered that applying invariance and equivariance constraints to the same feature layer causes a conflict. PUIR elegantly solves this by applying the equivariance constraint before interacting with the biological prior $\mathbb{O}$, and the invariance constraint after fusion.

Key Results

Modality Transfer (Image Synthesis)

PUIR significantly outperformed existing 2D and 3D generation methods (including popular GAN and diffusion-based models) on the BraTS23 dataset. For instance, in $T1 \rightarrow T2$ translation, PUIR achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.756, compared to ResViT's 25.578.

Missing Modality Segmentation

When evaluated on BraTS18 for brain tumor segmentation where certain modalities are missing during inference, PUIR demonstrated remarkable robustness. It consistently outperformed state-of-the-art methods like RFNet and mmFormer, particularly when the number of missing modalities was high (Missing Modalities = 3).

Heterogeneous Generalization (CT & PET)

On the AutoPET-II dataset, ablations proved the necessity of the learnable prior $\mathbb{O}$. Combining all constraints with the prior yielded the highest segmentation performance (Dice score of 48.20 vs. 43.45 from scratch).

Qualitative results of heterogeneous modality transfer between PET and CT.

Limitations and Future Work

Despite its strong performance, the authors note a few key limitations:

  • Instance-Level Matching Requirement: PUIR currently requires pre-training datasets where all modalities are perfectly matched at the instance level (i.e., all scans must belong to the same individual at the same time). This can be a very restrictive condition in real-world clinical settings.
  • Future Directions: The authors suggest exploring methods to achieve personalized invariance without relying on perfectly matched multi-modal datasets, alongside integrating advanced downstream tuning techniques like knowledge distillation.

Summary updated

Abstract

Variations in medical imaging modalities and individual anatomical differences pose challenges to cross-modality generalization in multi-modal tasks. Existing methods often concentrate exclusively on common anatomical patterns, thereby neglecting individual differences and consequently limiting their generalization performance. This paper emphasizes the critical role of learning individual-level invariance, i.e., personalized representation $\mathbb{X}_h$, to enhance multi-modality generalization under both homogeneous and heterogeneous settings. It reveals that mappings from individual biological profile to different medical modalities remain static across the population, which is implied in the personalization process. We propose a two-stage approach: pre-training with invariant representation $\mathbb{X}_h$ for personalization, then fine-tuning for diverse downstream tasks. We provide both theoretical and empirical evidence demonstrating the feasibility and advantages of personalization, showing that our approach yields greater generalizability and transferability across diverse multi-modal medical tasks compared to methods lacking personalization. Extensive experiments further validate that our approach significantly enhances performance in various generalization scenarios.
BibTeX
@misc{tan2024towards,
  author = {Zhaorui Tan and Xi Yang and Tan Pan and Tianyi Liu and Chen Jiang and Xin Guo and Qiufeng Wang and Anh Nguyen and Yuan Qi and Kaizhu Huang and Yuan Cheng},
  title = {{Towards a Universal 3D Medical Multi-modality Generalization via Learning Personalized Invariant Representation}},
  year = {2024},
  eprint = {2411.06106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.06106},
}
Figure