Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion

Zihao Wang, Herve Delingette

TL;DR

This paper introduces MIDiffusion, a novel zero-shot, unsupervised method for cross-modality image translation. It leverages a diffusion model guided by local-wise mutual information (LMI) to translate images between modalities without requiring paired data or retraining for new source domains, achieving high fidelity and semantic consistency.

What Problem Does This Paper Address?

Cross-modality data translation, the process of converting images from one modality (e.g., CT scans) to another (e.g., MRI scans), is crucial for leveraging existing medical imaging software and analysis tools across different data types. While deep generative models like GANs have shown promise, a significant challenge remains: zero-shot cross-modality data translation with high fidelity. This means translating images to a new modality without any prior training data from that specific source-target pair, while ensuring the translated image is both realistic (looks like a genuine image of the target modality) and faithful (preserves the anatomical structures and semantic meaning of the original image). Existing methods often struggle with this, either requiring paired datasets, extensive retraining for new modalities, or failing to balance realism and faithfulness, especially when the visual appearance of the modalities differs significantly.

Methods

The core of MIDiffusion lies in its innovative use of stochastic diffusion models combined with a novel local-wise mutual information (LMI) guidance mechanism.

  • Diffusion Models: The method is built upon score-matching-based generative models, which learn to generate data by iteratively denoising a noisy version of it. This is framed within a stochastic differential equation (SDE) framework, allowing for a generative process guided by a learned score function.
  • Local-wise Mutual Information (LMI) Guidance: This is the key novelty. Instead of relying on direct pixel-to-pixel mappings or adversarial losses, MIDiffusion uses LMI to measure the statistical similarity between local regions of the source and target images. This LMI acts as a "conditioner" for the diffusion process, guiding the denoising steps to ensure semantic consistency.
  • Zero-Shot and Unsupervised: Crucially, MIDiffusion is designed to be zero-shot, meaning it can translate to unseen modalities without retraining. It's also unsupervised, not requiring paired source-target image datasets. The LMI mechanism allows it to learn shared statistical features, bridging the gap between modalities without explicit domain mapping.
  • Differentiable LMI Layer: To make the LMI computation efficient within the iterative diffusion process, the authors introduce a differentiable LMI layer.

Datasets

The paper evaluates MIDiffusion on three publicly available cross-modality datasets:

  • Gold Atlas: CT, T1-weighted (T1w), and T2-weighted (T2w) MRI data.
  • CuRIOUS: T1w and FLAIR MRI scan pairs.
  • IXI dataset: PD-weighted and T1w MRI data.

These datasets cover various cross-modality translation tasks, such as CT to MR and vice-versa, and T1w to FLAIR and vice-versa.

Key Insights and Ideas

  • Mutual Information as a Universal Bridge: The paper demonstrates that local-wise mutual information can effectively capture shared statistical features between different imaging modalities, serving as a powerful, modality-agnostic bridge for translation.
  • Decoupling Source and Target: By relying on LMI rather than direct mapping, MIDiffusion avoids the need for retraining when the source domain changes. This is a significant advantage for practical applications where collecting extensive source data for every new modality is infeasible.
  • Diffusion Models for High-Fidelity Translation: The use of diffusion models, guided by LMI, allows for a more controlled and stable generation process, leading to improved faithfulness and realism compared to some GAN-based approaches.
  • Efficiency through Differentiable LMI: The development of a differentiable LMI layer is critical for integrating this guidance mechanism into the iterative diffusion process without prohibitive computational costs.

Key Results

  • Quantitative Performance: MIDiffusion outperforms baseline methods (CycleGAN, StyleGAN, SDEdit) across multiple metrics, including SSIM (for faithfulness), MSE, PSNR, and FID (for realism), on the Gold Atlas, CuRIOUS, and IXI datasets. Notably, it achieves higher semantic faithfulness and realism, particularly in zero-shot settings.
  • Qualitative Performance: Visual results (Figure 4) show that MIDiffusion generates translated images with better anatomical consistency and appearance similarity compared to baselines. It successfully preserves intricate anatomical structures, which is a common failure point for other methods.
  • Zero-Shot Advantage: The method demonstrates superior performance in zero-shot scenarios, especially when source domain data is scarce, outperforming even few-shot supervised methods like CycleGAN.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that MIDiffusion requires hundreds of iterative SDE solutions, leading to translation times of several seconds per image on a GPU.
  • Future Work: The authors suggest reducing the number of sampling steps without compromising translation fidelity as a key area for future research. This would make the method more practical for real-time applications.

Summary updated

Abstract

Cross-modality data translation has attracted great interest in image computing. Deep generative models (e.g., GANs) show performance improvement in tackling those problems. Nevertheless, as a fundamental challenge in image translation, the problem of Zero-shot-Learning CrossModality Data Translation with fidelity remains unanswered. This paper proposes a new unsupervised zero-shot-learning method named Mutual Information guided Diffusion cross-modality data translation Model (MIDiffusion), which learns to translate the unseen source data to the target domain. The MIDiffusion leverages a score-matching-based generative model, which learns the prior knowledge in the target domain. We propose a differentiable local-wise-MI-Layer (LMI) for conditioning the iterative denoising sampling. The LMI captures the identical cross-modality features in the statistical domain for the diffusion guidance; thus, our method does not require retraining when the source domain is changed, as it does not rely on any direct mapping between the source and target domains. This advantage is critical for applying cross-modality data translation methods in practice, as a reasonable amount of source domain dataset is not always available for supervised training. We empirically show the advanced performance of MIDiffusion in comparison with an influential group of generative models, including adversarial-based and other scorematching-based models
BibTeX
@misc{wang2023zero,
  author = {Zihao WANG and Herve Delingette},
  title = {{Zero-shot-Learning Cross-Modality Data Translation Through Mutual Information Guided Stochastic Diffusion}},
  year = {2023},
  eprint = {2301.13743},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.13743},
}
Figure