Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation

Yixin Chen, Lin Gao, Yajuan Gao, Rui Wang, Jingge Lian, Xiangxi Meng, Yanhua Duan, Leiying Chai, Hongbin Han, Zhaoping Cheng, Zhaoheng XiearXiv 2024 · DOI · arXiv

Introduction

Medical imaging is a fragmented landscape. A model trained to identify structures in a CT scan typically fails when applied to an MRI, and vice versa. While "universal models" aim to bridge this gap, they often encounter "modality interference"—a phenomenon where learning features from one scan type degrades performance on another. To address this, researchers from Peking University and affiliated institutions have developed the Modality-Projection Universal Model (MPUM). This framework is designed to handle the inherent variability of full-body medical imaging across multiple modalities (CT, MRI, and PET) using a novel projection strategy.

Problem Statement

The development of a single model capable of identifying every organ and lesion across any scan type faces two primary hurdles:

  1. Data Variability: Tissues appear fundamentally different across modalities; for instance, a liver is density-based in CT but relaxation-based in MRI.
  2. Modality Interference: Training a single model on mixed data often leads to conflicting optimization gradients, resulting in suboptimal performance across all modalities.

Existing foundation models frequently require labor-intensive task-specific fine-tuning. The MPUM aims to be a truly universal model that generalizes across tasks and modalities without requiring specialized "heads" for every new problem.

Methods: The Modality Projection Principle

The core innovation of MPUM is the Modality Projection Principle. Rather than treating each modality as a separate problem, the authors posit that every tissue (e.g., the spleen) possesses a high-dimensional latent feature—a "Platonic ideal" of that organ.

Overview of the MPUM framework, including training data, the modality-projection strategy, and clinical applications.

The model utilizes a Modality Projection Controller (MPC) to project these latent features into specific modality spaces (CT, MRI, or PET) via a projection matrix. Key technical features include:

  • Dynamic Kernels: The MPC employs a Multi-Layer Perceptron (MLP) to generate custom convolutional kernels on the fly, tailored to the input image's modality.
  • Dual-Branch Architecture: Each network block contains two paths: a traditional branch for modality-invariant features (e.g., global contours) and a controller-based branch for modality-specific nuances (e.g., texture).

Datasets and Training

The researchers compiled an extensive multi-modal training suite comprising:

  • FDG-PET-CT: 533 subjects for whole-body metabolic and structural data.
  • DAP Atlas & TotalSegmentatorMRI: Providing broad anatomical labels for 142 and 59 structures, respectively.
  • BrainAtlas: 30 subjects for detailed neuroanatomy.
  • Instance2022: 100 CT scans for Intracranial Hemorrhage (ICH) fine-tuning.
  • Pediatric Epilepsy: A custom dataset of 77 subjects for metabolic association analysis.

In total, the model is capable of identifying 215 categories, including 132 non-brain and 83 brain regions.

Key Insights

  • Mitigating Interference: By using projection matrices, the model shares knowledge about organ boundaries while maintaining separate filters for textures. This prevents gradient conflict during training.
  • The Brain-Body Axis: Simultaneous segmentation of the brain and body across PET and CT allowed researchers to identify significant metabolic correlations between brain regions (e.g., the Pallidum) and body structures (e.g., T1-T12 vertebrae) in epilepsy patients, demonstrating the model's utility as a discovery tool.
  • Interpretability: Visualizing saliency maps revealed that shallow layers focus on global contours, while deeper layers progressively refine the specific region of interest.

Key Results

The MPUM demonstrated state-of-the-art performance, outperforming top-tier models such as STUNet and CDUM.

Performance comparison of MPUM against other strategies across CT, MR, and PET modalities for both body and brain segmentation.
  • Segmentation Accuracy: In CT body segmentation, MPUM achieved a Dice score of 0.8517.
  • Clinical Impact: In a study of 28 intracranial hemorrhage cases, MPUM assistance increased the diagnostic accuracy of junior radiologists from 71.4% to 96.4%.
  • Operational Efficiency: The model is deployment-ready, requiring 16GB of VRAM and processing a whole-body CT in 250–450 seconds.

Limitations and Future Work

While highly effective, the computational overhead for controller-based kernels may increase as the model scales to thousands of lesion types. Furthermore, integrating such AI into high-pressure clinical workflows remains a logistical challenge. Future research will focus on continual training frameworks to allow the model to learn from new data without forgetting previous tasks.

Summary updated

BibTeX
@misc{chen2024modality,
  author = {Yixin Chen and Lin Gao and Yajuan Gao and Rui Wang and Jingge Lian and Xiangxi Meng and Yanhua Duan and Leiying Chai and Hongbin Han and Zhaoping Cheng and Zhaoheng Xie},
  title = {{Modality-Projection Universal Model for Comprehensive Full-Body Medical Imaging Segmentation}},
  year = {2024},
  eprint = {2412.19026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.19026},
}
Figure