TL;DR
MIDiffusion is a novel unsupervised, zero-shot framework for cross-modality image translation (e.g., converting a CT scan to an MRI). By guiding a score-based diffusion model with a differentiable Local-wise Mutual Information (LMI) layer, the method achieves high anatomical faithfulness without ever seeing the source modality during training.
What Problem Does This Paper Address?
In medical imaging, we often need to translate data between modalities—such as generating a synthetic MRI from a CT scan to use existing brain parcellation software. This is challenging for two main reasons:
- Data Scarcity: Supervised methods require large, perfectly aligned (paired) datasets of both modalities, which are rarely available.
- The Fidelity Trade-off: Generative models must balance realism (does the output look like the target modality?) and faithfulness (does the output preserve the original anatomy?).
Existing zero-shot methods like SDEdit use noise perturbation for guidance, but this often fails when the source and target domains look significantly different (e.g., brightfield vs. darkfield microscopy). MIDiffusion aims to bridge this gap by using statistical features that remain consistent across modalities.
Methods
The authors propose MIDiffusion, which combines score-based generative models (diffusion) with a novel guidance mechanism.
- Score-Based Diffusion: The model is trained only on the target domain to learn its data distribution. It learns to reverse a process that gradually adds Gaussian noise to an image until it becomes pure white noise.
- LMI Guidance: To ensure the translated image matches the source image's anatomy, the authors introduce a Local-wise Mutual Information (LMI) layer. Mutual Information (MI) is a statistical measure that quantifies how much information is shared between two variables. Crucially, MI is modality-invariant; it doesn't care about pixel intensity, only about statistical dependencies.
- Differentiable LMI Layer: Standard MI is difficult to optimize directly. The authors developed a differentiable LMI layer using kernel density estimation, allowing it to be embedded directly into the iterative denoising process of the diffusion model.
During the reverse SDE (sampling) step, the LMI layer calculates the statistical similarity between the source image and the currently sampled image, providing a "gradient" that nudges the diffusion process toward anatomical consistency.
Datasets
The model was evaluated on three public medical imaging datasets:
- Gold Atlas: CT and T1-weighted/T2-weighted MRI pairs of the male pelvis.
- CuRIOUS: T1-weighted and FLAIR MRI scans of subjects with low-grade glioma.
- IXI: PD-weighted and T1-weighted MRI scans from normal subjects.
Key Insights and Ideas
- Statistical Homogeneity: The core "aha" moment is that while pixel values change drastically between CT and MRI, the underlying anatomical structures share identical local statistical features. LMI is the perfect tool to capture these shared features.
- Zero-Shot Practicality: Because the model only needs to see the target domain during training, it can translate any unseen source modality into that target domain at test time without retraining.
- Guidance over Mapping: Unlike GANs that try to learn a direct mapping ($X \to Y$), MIDiffusion uses the source image as a "conditioner" for a generative process that already knows what the target domain should look like.
Key Results
MIDiffusion was compared against CycleGAN (few-shot supervised), StyleGAN (zero-shot inversion), and SDEdit (zero-shot diffusion).
- Superior Faithfulness: MIDiffusion consistently achieved higher SSIM (Structural Similarity) and PSNR (Peak Signal-to-Noise Ratio) compared to other zero-shot methods, indicating it preserves anatomy much better.
- Outperforming Few-Shot GANs: In the GoldAtlas and CuRIOUS datasets, the zero-shot MIDiffusion actually outperformed CycleGAN even when the latter was given access to a small amount of source domain data (2-6%).
- Realism: While SDEdit sometimes achieved lower FID scores (a measure of image quality), it often failed to maintain anatomical faithfulness, whereas MIDiffusion maintained a strong balance of both.
Limitations and Future Work
The primary limitation acknowledged is computational cost. Like most diffusion models, MIDiffusion requires hundreds of iterative steps to generate a single image, taking dozens of seconds on a modern GPU. The authors suggest that reducing the number of sampling steps without losing fidelity is a key direction for future research. Additionally, while the method is demonstrated on 2D slices, extending it efficiently to 3D volumes remains a practical challenge.
Summary updated