Diffusion based Zero-shot Medical Image-to-Image Translation for Cross Modality Segmentation

Zihao Wang, Yingyu Yang, Yuzhou Chen, Tingting Yuan, Maxime Sermesant, Herve Delingette, Ona WuarXiv 2024 · DOI · arXiv

Introduction

High-quality medical image segmentation tools are typically optimized for specific modalities, such as T1-weighted (T1w) MRI. Adapting these tools to a different modality, like Proton Density-weighted (PDw) images, often necessitates training new models from scratch, a process constrained by the limited availability of labeled data. While Generative Adversarial Networks (GANs) and recent diffusion models like SDEdit attempt to bridge this gap, they frequently require exposure to both domains during training or struggle to maintain anatomical integrity in zero-shot scenarios.

To address these challenges, this paper introduces LMIDiffusion, a novel unsupervised framework that utilizes diffusion models to translate medical images between modalities without prior exposure to the source modality during training. By employing Locale-based Mutual Information (LMI) as a guidance mechanism, the model preserves anatomical structures while adapting to new modalities, enabling the use of existing segmentation tools on previously "unseen" data types.

Methods

LMIDiffusion treats cross-modality translation as a guided denoising process. The framework uses a score-matching diffusion backbone trained exclusively on the target modality (e.g., T1w images).

Overview of the LMIDiffusion framework using LMI guidance for zero-shot translation.

The core innovation is the use of Locale-based Mutual Information (LMI). Unlike pixel-to-pixel mappings, LMI leverages the fact that local statistical relationships—such as edges and boundaries—remain consistent across different modalities. During the sampling phase, the LMI between the source image and the current noisy state guides the diffusion process. This ensures the generated image adopts the style of the target modality while strictly adhering to the anatomical structure of the source. This general statistical approach grants the model zero-shot capability, as it does not require retraining when the source modality changes.

Results and Insights

The researchers evaluated LMIDiffusion using the IXI dataset, translating PDw images to T1w for subsequent segmentation using a K-Means algorithm trained only on T1w data.

Qualitative comparison of LMIDiffusion against GAN and diffusion baselines.

LMIDiffusion significantly outperformed both GAN-based and other diffusion-based baselines across key quantitative metrics:

Metric CycleGAN StyleGAN SDEdit LMIDiffusion
Dice Score 0.85 ± 0.07 0.66 ± 0.10 0.82 ± 0.06 0.88 ± 0.05
PSNR 18.88 ± 1.26 10.15 ± 1.49 15.96 ± 1.54 20.22 ± 1.43
SSIM 0.27 ± 0.04 0.06 ± 0.03 0.50 ± 0.06 0.69 ± 0.06

The Dice score of 0.88 indicates that the translated images were accurate enough for a segmentation tool that had never seen a PDw image. Qualitatively, LMIDiffusion retained sharp anatomical details that were often blurred or distorted by CycleGAN or StyleGAN.

Key insights from the study include:

  • Statistical Homogeneity: LMI captures the semantic essence of boundaries without needing paired training data.
  • Anatomy Preservation: LMI guidance acts as a rigid structural constraint, preventing the hallucinations common in standard diffusion-based editing.
  • Clinical Practicality: The model's adaptability to different scanners and protocols without retraining makes it highly suitable for real-world clinical settings.

Conclusion and Future Work

While LMIDiffusion offers superior structural preservation and zero-shot flexibility, its iterative sampling process is computationally slower than the single-pass inference of GANs. Future research will focus on refining the guidance mechanism, potentially incorporating few-shot learning, and exploring the model's performance on complex pathological images, such as tumors, where appearance changes drastically across modalities.

Summary updated

Abstract

Cross-modality image segmentation aims to segment the target modalities using a method designed in the source modality. Deep generative models can translate the target modality images into the source modality, thus enabling cross-modality segmentation. However, a vast body of existing cross-modality image translation methods relies on supervised learning. In this work, we aim to address the challenge of zero-shot learning-based image translation tasks (extreme scenarios in the target modality is unseen in the training phase). To leverage generative learning for zero-shot cross-modality image segmentation, we propose a novel unsupervised image translation method. The framework learns to translate the unseen source image to the target modality for image segmentation by leveraging the inherent statistical consistency between different modalities for diffusion guidance. Our framework captures identical cross-modality features in the statistical domain, offering diffusion guidance without relying on direct mappings between the source and target domains. This advantage allows our method to adapt to changing source domains without the need for retraining, making it highly practical when sufficient labeled source domain data is not available. The proposed framework is validated in zero-shot cross-modality image segmentation tasks through empirical comparisons with influential generative models, including adversarial-based and diffusion-based models.
BibTeX
@misc{wang2024diffusion,
  author = {Zihao Wang and Yingyu Yang and Yuzhou Chen and Tingting Yuan and Maxime Sermesant and Herve Delingette and Ona Wu},
  title = {{Diffusion based Zero-shot Medical Image-to-Image Translation for Cross Modality Segmentation}},
  year = {2024},
  eprint = {2404.01102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.01102},
}
Figure