Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images

Suhyun Ahn, Wonjung Park, Ji‐Hoon Cho, Sean Park, Jinah Park

TL;DR

This paper introduces the Volumetric Conditioning Module (VCM), a novel, lightweight module designed to control large pretrained diffusion models for generating 3D medical images. VCM enables precise conditional generation, even with limited data, and demonstrates effectiveness across tasks like super-resolution and image translation, addressing the challenges of high computational costs and data scarcity in medical imaging.

What Problem Does This Paper Address?

Generating high-quality 3D medical images is crucial for various applications, but training diffusion models from scratch is computationally expensive and requires vast amounts of data, which are often scarce and privacy-sensitive in the medical domain. While spatial control methods have shown promise in natural image generation by guiding pretrained models with additional conditions, their application to 3D medical imaging remained unexplored. This paper aims to bridge this gap by developing a specialized spatial control method, VCM, that can effectively leverage existing large diffusion models for 3D medical image synthesis under various conditions, even with limited training data.

Methods

The core contribution is the Volumetric Conditioning Module (VCM). VCM is designed as a lightweight, asymmetric U-Net architecture that injects control signals into a pretrained diffusion model (specifically BrainLDM in their experiments) at each denoising step. Key aspects of VCM include:

  • Asymmetric U-Net: Features a deeper encoder than decoder to effectively capture complex 3D spatial condition information.
  • Time Conditioning: Incorporates time embedding layers to provide fine-grained guidance throughout the diffusion process.
  • Modulation Approach: VCM modulates the diffusion model's output (noise prediction) using learned parameters, rather than directly fusing features like some prior methods. This is achieved by outputting shift ($\gamma_t$) and scale ($\beta_t$) tensors that modify the predicted noise.
  • Freezing Backbone: The large pretrained diffusion model's parameters are frozen, allowing VCM to be trained efficiently with a smaller number of parameters.
  • Modality Latent Dropout: For multimodal training, a regularization technique is introduced to prevent over-reliance on any single condition, improving robustness and enabling plug-and-play conditioning.

Datasets

The primary dataset used is from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database, comprising 604 healthy T1w Brain MRI scans. For generating spatial conditions, segmentation masks for the lateral ventricle (LV) and brain were extracted using SynthSeg. Partial images of the skull were also used. Experiments were conducted using subsets of this data, specifically:

  • Extremely small datasets (10 samples)
  • Small datasets (50 samples)
  • Large datasets (500 samples)

For super-resolution tasks, 1 mm³ T1w MRI scans were downsampled to 4 mm and 8 mm resolutions to create low-resolution (LR) inputs.

Key Insights and Ideas

  • Spatial Control is Viable for 3D Medical Imaging: The paper demonstrates that spatial control methods, previously successful in 2D natural images, can be effectively adapted and are highly beneficial for 3D medical image generation.
  • Data Efficiency is Crucial: VCM's performance, especially on small datasets, highlights the importance of efficient methods in medical imaging where data is often limited.
  • Lightweight Modules Can Be Powerful: VCM, despite its significantly smaller parameter count compared to some other methods, achieves competitive or superior results, emphasizing the impact of tailored architecture design.
  • Modality Latent Dropout Enhances Multimodal Training: This regularization technique proves effective in preventing model collapse and improving stability during multimodal synthesis.
  • Qualitative Evaluation is Paramount: The paper stresses that quantitative metrics alone are insufficient for evaluating synthetic medical images, and visual inspection is critical, especially for detecting anatomically incorrect or destroyed outputs.

Key Results

  • Alignment Scores: VCM consistently achieved high alignment scores (e.g., DSC, ASSD) across various dataset sizes, often outperforming other methods, especially on smaller datasets. For instance, on the extremely small dataset (10 samples), VCM showed the highest alignment scores in DSC and ASSD.
  • Parameter Efficiency: VCM has a significantly smaller parameter count (e.g., 45.19M) compared to methods like ControlNet (193.88M) and T2I-Adapter (223.41M), while achieving comparable or better results.
  • Training Efficiency: VCM requires less training data and computational resources. The paper reports training times in GPU days, showing VCM to be competitive or more efficient than other methods.
  • Super-Resolution: In the challenging 8 mm axial super-resolution task, VCM produced the most anatomically plausible results, maintaining brain structures and details, outperforming LIIF-3D and InverseSR in qualitative and quantitative metrics.
  • Image Translation: VCM successfully synthesized T1w MRI scans from T2 and FLAIR inputs, producing clearer images without grid artifacts, outperforming HyperGAE.
  • Robustness to Outliers: VCM demonstrated robustness in handling out-of-distribution conditions, such as abnormal LV masks, generating plausible images with correct localization.

Limitations and Future Work

  • Evaluation Metrics: The authors acknowledge that standard metrics like FID and LPIPS, while useful, are insufficient for fully evaluating the quality and anatomical correctness of synthetic medical images. They suggest the need for more innovative and effective evaluation metrics.
  • Anatomically Incorrect Conditions: While VCM can handle some out-of-distribution conditions, severely abnormal inputs (e.g., destroyed skull and brain stem) can lead to anatomically incorrect outputs, highlighting the importance of careful use and evaluation of spatial control methods.
  • Future Work: The paper suggests that VCM, being efficient and effective, is well-suited for adoption in various downstream tasks in medical imaging. Further exploration of its applications and the development of more advanced evaluation metrics are implied.

Summary updated

Abstract

Spatial control methods using additional modules on pretrained diffusion models have gained attention for enabling conditional generation in natural images. These methods guide the generation process with new conditions while leveraging the capabilities of large models. They could be beneficial as training strategies in the context of 3D medical imaging, where training a diffusion model from scratch is challenging due to high computational costs and data scarcity. However, the potential application of spatial control methods with additional modules to 3D medical images has not yet been explored. In this paper, we present a tailored spatial control method for 3D medical images with a novel lightweight module, Volumetric Conditioning Module (VCM). Our VCM employs an asymmetric U-Net architecture to effectively encode complex information from various levels of 3D conditions, providing detailed guidance in image synthesis. To examine the applicability of spatial control methods and the effectiveness of VCM for 3D medical data, we conduct experiments under single- and multimodal conditions scenarios across a wide range of dataset sizes, from extremely small datasets with 10 samples to large datasets with 500 samples. The experimental results show that the VCM is effective for conditional generation and efficient in terms of requiring less training data and computational resources. We further investigate the potential applications for our spatial control method through axial super-resolution for medical images. Our code is available at \url{https://github.com/Ahn-Ssu/VCM}
BibTeX
@inproceedings{ahn2025volumetric,
  author = {Suhyun Ahn and Wonjung Park and Ji‐Hoon Cho and Sean Park and Jinah Park},
  title = {{Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00019},
}
Figure