Personalize to generalize: Towards a universal medical multi-modality generalization through personalization

Zhaorui Tan, Xi Yang, Tan Pan, Tianyi Liu, Chen Jiang, Xin Guo, Qiufeng Wang, Anh Nguyen, Yuan Qi, Kaizhu Huang, Yuan ChengSubmission to ICLR 2025 · Desk rejected

TL;DR

Rather than stripping away individual differences to find generic anatomical features, this paper proposes learning a personalized invariant representation ($\mathbb{X}_h$) that captures patient-specific biology across medical imaging modalities. By combining a learnable global biological prior with decomposition, equivariance, and invariance constraints, the model effectively shares knowledge across both structural (MRI, CT) and functional (PET) imaging, outperforming current state-of-the-art methods in cross-modality synthesis and missing-modality segmentation.

What Problem Does This Paper Address?

In clinical healthcare, doctors rely on multiple imaging modalities—such as structural scans (MRI, CT) that show physical anatomy and functional scans (PET) that measure biochemical activity. However, in practice, patients often lack access to complete sets of scans due to high costs, radiation risks, or logistical barriers.

Generalizing across modalities is notoriously difficult because imaging physics differ drastically—especially between structural and functional modalities (termed heterogeneous generalization). Most existing multi-modal generalization techniques attempt to learn a domain-invariant representation ($Z$) that captures shared class-level or population-level anatomy. In doing so, they discard patient-specific nuances and modality-unique features, making it impossible to reconstruct individual modalities accurately or generalize robustly when key modalities are missing.

Methods

The authors introduce the $\mathbb{X}_h$ Hypothesis: for any individual $h$, there exists a core personalized invariant biological representation $\mathbb{X}_h$ that is invariant to modality $\mathcal{M}$ and spatial transformation $\Phi$, yet contains enough information to decompose into any specific modality $X_h^i$.

Input Scan (Modality i) ──► Encoder ──► Feature z_h^i ──► Cross-Attention with Prior O ──► Fused Representation X_h^i'
                                                                                                    │
                                 ┌──────────────────────────────────────────────────────────────────┴──┐
                                 ▼                                  ▼                                  ▼
                         Invariance Loss                  Decomposition Loss                  Equivariance Loss
                   (Align X_h^i' across modalities)   (Reconstruct all modalities X_h)     (Predict SO(3) rotations)

To approximate $\mathbb{X}_h$ in practice:

  • Global Biological Prior ($\mathbb{O}$): A learnable tensor acts as an external biological memory bank. Given latent feature $z_h^i$ from an accessible modality, cross-attention retrieves missing cross-modal biological context from $\mathbb{O}$, which is then fused via convolution into an estimated representation $X_h^{i\prime}$.
  • Three Joint Pre-training Constraints:
  • Equivariance Constraint ($\mathcal{L}_{\text{equ}}$): Enforces intermediate representations to encode 3D spatial transformations (predicting discrete $SO(3)$ volume rotations).
  • Invariance Constraint ($\mathcal{L}_{\text{inv}}$): Forces the fused patient representations $X_h^{i\prime}$ across different modalities and orientations to align with each other.
  • Decomposition Constraint ($\mathcal{L}_{\text{decom}}$): A decoder uses the fused invariant representation along with shallow encoder features to reconstruct all target modalities simultaneously.
  • Contrastive Learning Baseline ($\mathcal{L}_{\text{contrast}}$): Standard sub-volume contrastive loss applied to positive and negative volume views.
  • Downstream Adaptation: The pre-trained SwinUNETR backbone is fine-tuned on downstream tasks (segmentation or generation) using standard task loss ($\mathcal{L}_{\text{ori}}$) combined with the invariance constraint ($\mathcal{L}_{\text{inv}}$).

Datasets

  • BraTS 2023: 4 structural MRI sequences (T1, T1ce, T2, FLAIR) from 219 individuals, used for multi-modal brain tumor image translation.
  • BraTS 2018: 4 structural MRI sequences used to evaluate missing-modality brain tumor segmentation under varying combinations of missing modalities.
  • AutoPET-II: Paired whole-body FDG-PET and CT scans with tumor lesion annotations, used for heterogeneous modality transfer (PET $\leftrightarrow$ CT) and multi-modal tumor segmentation.
  • HNSCC & NSCLC (from TCIA): Head and Neck Squamous Cell Carcinoma (training) and Non-Small Cell Lung Cancer (testing) datasets containing non-attenuation-corrected PET (NAC-PET), attenuation-corrected PET (AC-PET), and CT scans for evaluating cross-domain, heterogeneous-to-homogeneous transfer.

Key Insights and Ideas

  • Personalize to generalize: Standard domain-invariant models collapse patient-specific context in favor of population-level averages. Retaining patient-specific features across modalities actually improves transferability rather than hindering it.
  • Why global biological prior $\mathbb{O}$ matters: A single modality (e.g., FLAIR MRI or CT) inherently lacks certain tissue contrast or metabolic data. Cross-attending into a learnable prior $\mathbb{O}$ enables the model to complete missing biological context before enforcing invariance.
  • Separating equivariance and invariance: The authors discovered that geometric equivariance (preserving orientation) and representation invariance cannot simply be applied to the same latent space without degrading performance. Equivariance must be applied to initial single-modality features, and invariance enforced after querying the prior $\mathbb{O}$.

Key Results

  • Modality Synthesis (BraTS23): Outperformed state-of-the-art 2D and 3D generative frameworks (including diffusion models like MD-Diff and MS-SPADE). For T1 $\rightarrow$ T2 translation, the method achieved 30.76 dB PSNR and 0.944 SSIM (vs. MS-SPADE's 25.82 dB / 0.904 SSIM).
  • Missing Modality Segmentation (BraTS18): In the toughest scenario where 3 out of 4 modalities were missing, the method surpassed mmFormer across all tumor regions:
  • Tumor Core: 74.51 Dice (vs. mmFormer's 71.45)
  • Enhancing Tumor: 60.19 Dice (vs. mmFormer's 48.86)
  • Whole Tumor: 85.17 Dice (vs. mmFormer's 80.45)
  • Heterogeneous Segmentation (AutoPET-II): Achieved 48.20 Dice on FDG-PET/CT lesion segmentation, noticeably higher than baseline SwinUNETR without pre-training (43.45) and with standard self-supervised pre-training (44.06).
  • Cross-Domain Generalization: In cross-domain transfer (HNSCC $\rightarrow$ NSCLC) from NAC-PET to CT, the method improved SSIM from 0.4476 (SwinUNETR) to 0.4744 and PSNR from 22.51 dB to 22.78 dB.

Limitations and Future Work

  • Strict Data Matching Requirement: The pre-training objective relies heavily on fully matched, co-registered, multi-modal scans from the exact same patient visit—a condition that is difficult to meet in routine clinical settings.
  • Scarcity of Functional Datasets: Open-source paired multi-modal datasets involving functional imaging (PET, SPECT) remain limited, constraining the scale of heterogeneous pre-training.
  • Alternative Transfer Mechanisms: The tuning phase currently uses straightforward fine-tuning; the authors suggest investigating more advanced downstream strategies like knowledge distillation to maximize transfer gains.

Summary updated

Abstract

Personalized medicine is a groundbreaking healthcare framework for the $21^{st}$ century, tailoring medical treatments to individuals based on unique clinical characteristics, including diverse medical imaging modalities. These modalities differ significantly due to distinct underlying imaging principles, creating substantial challenges for generalization in multi-modal medical image tasks. Previous methods addressing multi-modal generalization rarely consider personalization, primarily focusing on common anatomical information. This paper aims to connect multi-modal generalization with the concept of personalized medicine. Specifically, we propose a novel approach to derive a tractable form of the underlying personalized invariant representation $\mathbb{X}_h$ using individual-level constraints and a learnable biological prior. We demonstrate that learning a personalized $\mathbb{X}_h$ is both feasible and beneficial, as this representation proves highly generalizable and transferable across various multi-modal medical tasks. Our method is rigorously validated on medical imaging modalities emphasizing both physical structure and functional information, encompassing a range of tasks that require generalization. Extensive experimental results consistently show that our approach significantly improves performance across diverse scenarios, confirming its effectiveness.
BibTeX
@misc{tan2024personalize,
  author = {Zhaorui Tan and Xi Yang and Tan Pan and TIANYI LIU and Chen Jiang and Xin Guo and Qiufeng Wang and Anh Nguyen and Yuan Qi and Kaizhu Huang and Yuan Cheng},
  title = {{Personalize to generalize: Towards a universal medical multi-modality generalization through personalization}},
  year = {2024},
  eprint = {2411.06106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.06106},
}
Figure