Diffusion Deformable Model for 4D Temporal Medical Image Generation

Boah Kim, Jong Chul Ye

Abstract

Temporal volume images with 3D+t (4D) information are essential in medical imaging for analyzing temporal dynamics and capturing disease progression. However, generating realistic temporal sequences, such as cardiac volume data, remains a significant challenge. This paper presents the Diffusion Deformable Model (DDM), a novel deep learning framework that generates intermediate temporal volumes between source and target volumes. By combining a denoising diffusion probabilistic model with a deformation module, DDM learns spatial deformation information and provides a latent code for generating intermediate frames along a continuous geodesic path. Our results on 4D cardiac MR images demonstrate that DDM preserves image topology and outperforms existing deformation methods in generating high-quality temporal volumes.

Introduction

Capturing the progression of anatomical changes—such as the contraction of a heart or the growth of a tumor—is vital for accurate clinical diagnosis. While 4D MRI provides these insights, the modality is limited by long scan times, which often make it difficult to capture rapid motion with high resolution.

Artificial intelligence offers a potential solution by synthesizing intermediate frames to "fill the gaps" in temporal sequences. However, existing approaches face two primary hurdles:

  1. Generative Hallucinations: Generative Adversarial Networks (GANs) often introduce artificial features or artifacts that do not exist in the patient's actual anatomy.
  2. Anatomical Inconsistency: Standard registration methods, which warp one image to another, often struggle to produce realistic intermediate steps, sometimes resulting in anatomically impossible transitions or broken topologies.

The Diffusion Deformable Model (DDM) addresses these issues by leveraging the realism of diffusion models while maintaining the structural integrity of traditional image registration.

Methodology

The DDM architecture is split into two specialized components: the Diffusion Module and the Deformation Module.

The Diffusion Module

Based on Denoising Diffusion Probabilistic Models (DDPM), this module is tasked with learning the relationship between a "source" volume (e.g., end-diastole) and a "target" volume (e.g., end-systole). Instead of generating pixels directly, the diffusion process extracts a highly informative latent code ($c$). This code encapsulates the complex spatial information required to transform the source into the target.

The Deformation Module

To ensure anatomical consistency, the model does not generate new pixels from scratch. Instead, the deformation module takes the latent code and the source image to estimate deformation fields ($\phi$). These fields act as digital "stretching" instructions, warping the source image into the target shape using a Spatial Transformation Layer (STL).

Continuous Trajectory Generation

The "secret sauce" of DDM is its ability to perform linear interpolation within the latent space. Once the model is trained, the latent code can be scaled by a factor ($\gamma$) between 0 and 1. By feeding a scaled code (e.g., $0.5c$) into the deformation module, the system generates a perfectly intermediate frame. This allows for the synthesis of a continuous "geodesic path" of motion, ensuring smooth transitions that preserve the topology of the original tissue.

Experimental Setup

The model was evaluated using the Automated Cardiac Diagnosis Challenge (ACDC) dataset, which contains 100 4D cardiac MRI scans.

  • Data Split: 90 scans were used for training and 10 for testing.
  • Preprocessing: All images were resampled to a consistent voxel spacing and cropped to a resolution of $128 \times 128 \times 32$.
  • Baselines: DDM was compared against popular registration baselines, including VoxelMorph (VM) and its diffeomorphic variant (VM-diff).

Key Results

DDM demonstrated superior performance across both image quality and anatomical accuracy metrics:

  • Image Quality: DDM achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.725 dB, outperforming both VM (30.562) and VM-diff (29.481).
  • Anatomical Accuracy: Using Dice scores to measure how well the generated anatomy matches ground truth segmentations, DDM reached 0.802, a statistically significant improvement over the baseline models.
  • Inference Efficiency: While the diffusion training process is computationally intensive (approximately 6 hours), generating a new frame at inference time takes only 0.456 seconds. This speed makes the model viable for real-time clinical visualization and review.

Discussion and Limitations

By using the diffusion "score function" as a feature extractor for motion rather than just appearance, DDM captures non-linear anatomical dynamics more effectively than methods that simply scale motion vectors. Because it relies on warping (deformation) rather than pixel-wise generation, it is physically impossible for the model to add random artifacts; the heart tissue remains connected and anatomically plausible.

However, some limitations remain:

  • Dataset Diversity: The current study is limited to the ACDC dataset. Future validation on more diverse pathologies and different organs (such as the lungs) is required.
  • Interpolation vs. Extrapolation: The model currently requires known endpoints (source and target). Future iterations could explore "extrapolating" motion to predict future states from a single starting volume.
  • Hyper-parameter Sensitivity: The balance between diffusion loss and deformation loss is critical; improper weighting can lead to a drop in registration accuracy.

Conclusion

The Diffusion Deformable Model represents a significant advancement in 4D medical image synthesis. By combining the generative power of diffusion models with the topological constraints of deformation modules, DDM provides a robust framework for generating realistic, continuous temporal sequences in cardiac imaging.

Summary updated

Abstract

Temporal volume images with 3D+t (4D) information are often used in medical imaging to statistically analyze temporal dynamics or capture disease progression. Although deep-learning-based generative models for natural images have been extensively studied, approaches for temporal medical image generation such as 4D cardiac volume data are limited. In this work, we present a novel deep learning model that generates intermediate temporal volumes between source and target volumes. Specifically, we propose a diffusion deformable model (DDM) by adapting the denoising diffusion probabilistic model that has recently been widely investigated for realistic image generation. Our proposed DDM is composed of the diffusion and the deformation modules so that DDM can learn spatial deformation information between the source and target volumes and provide a latent code for generating intermediate frames along a geodesic path. Once our model is trained, the latent code estimated from the diffusion module is simply interpolated and fed into the deformation module, which enables DDM to generate temporal frames along the continuous trajectory while preserving the topology of the source image. We demonstrate the proposed method with the 4D cardiac MR image generation between the diastolic and systolic phases for each subject. Compared to the existing deformation methods, our DDM achieves high performance on temporal volume generation.
BibTeX
@inproceedings{kim2022diffusion,
  author = {Boah Kim and Jong Chul Ye},
  title = {{Diffusion Deformable Model for 4D Temporal Medical Image Generation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=z-EcRRbejvg},
}
Figure