Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images

Park Jinah, Suhyun Ahn, Wonjung Park, Jihoon Cho, Jinah Park

Introduction

Diffusion models have emerged as a powerful paradigm for image generation, yet their application in 3D medical imaging is often constrained by high computational costs and the scarcity of large-scale annotated datasets. In clinical settings, precise spatial control is paramount—for instance, synthetic images must strictly adhere to provided anatomical masks or partial scans. While 2D control mechanisms like ControlNet have been successful for natural images, they are not inherently optimized for the volumetric nature of medical data.

This paper introduces the Volumetric Conditioning Module (VCM), a lightweight, asymmetric plug-in designed to provide precise spatial control over pretrained 3D diffusion models. By freezing the base model and training only the VCM, researchers can achieve high-fidelity results even with extremely limited data.

Methodology

The VCM is designed as an architectural "sidecar" to a frozen 3D diffusion backbone (e.g., BrainLDM). This modular approach preserves the extensive anatomical knowledge of the pretrained model while introducing specific spatial guidance.

Key Innovations

  • Asymmetric U-Net Architecture: VCM utilizes a deep encoder to capture complex 3D spatial cues and a shallower decoder to map these features into the base model's latent space. This asymmetry is optimized for parameter efficiency and 3D data compression.
  • Feature Modulation: Instead of simple additive coupling, VCM predicts modulation parameters ($\gamma$ and $\beta$) to scale and shift the internal feature maps of the base model, allowing for more granular and effective guidance.
  • Modality Latent Dropout: To handle diverse conditioning inputs such as masks and partial images, a dropout strategy is employed during training. This ensures the model remains robust even when certain inputs are missing during inference.
  • Time-Conditioning: The module incorporates diffusion timestep embeddings, allowing it to adjust the strength of spatial guidance as the generation process progresses from noise to a detailed reconstruction.

Experimental Results

The VCM was evaluated using the ADNI database (604 T1-weighted brain MRI scans) across three training regimes: 10, 50, and 500 scans. The module was tested on tasks including semantic map-to-MRI synthesis, partial image completion, and axial super-resolution.

Overview of VCM performance across various conditioning tasks.

Data Efficiency and Alignment

VCM demonstrated exceptional performance in data-scarce environments. With only 10 training samples, it achieved a Dice score of 0.877 for ventricle-guided synthesis, significantly outperforming standard Latent Diffusion Models (LDMs) trained from scratch (0.656). In larger-scale tests (500 samples), VCM achieved a Hausdorff Distance (HD95) of 13.346, outperforming the T2I-Adapter (16.231) and matching the performance of the much heavier ControlNet (12.000) while being significantly more efficient.

Axial Super-Resolution

A key application of VCM is axial super-resolution, where thick-slice scans are enhanced to thin-slice quality. By treating this as a spatial conditioning problem, VCM achieved a PSNR of 27.518 for 8x super-resolution, vastly outperforming specialized methods like SynthSR (19.020). This success highlights VCM's ability to leverage the rich anatomical priors already present in the pretrained diffusion backbone.

Conclusion

The Volumetric Conditioning Module provides a robust and efficient framework for steering 3D medical diffusion models. Its ability to achieve high-fidelity results with as few as 10 scans makes it a valuable tool for medical research where data is limited. Future work will focus on extending VCM to other anatomical regions and evaluating its performance on pathological cases.

Summary updated

Abstract

Spatial control methods using additional modules on pretrained diffusion models have gained attention for enabling conditional generation in natural images. These methods guide the generation process with new conditions while leveraging the capabilities of large models. They could be beneficial as training strategies in the context of 3D medical imaging, where training a diffusion model from scratch is challenging due to high computational costs and data scarcity. However, the potential application of spatial control methods with additional modules to 3D medical images has not yet been explored. In this paper, we present a tailored spatial control method for 3D medical images with a novel lightweight module, Volumetric Conditioning Module (VCM). Our VCM employs an asymmetric U-Net architecture to effectively encode complex information from various levels of 3D conditions, providing detailed guidance in image synthesis. To examine the applicability of spatial control methods and the effectiveness of VCM for 3D medical data, we conduct experiments under single- and multimodal conditions scenarios across a wide range of dataset sizes, from extremely small datasets with 10 samples to large datasets with 500 samples. The experimental results show that the VCM is effective for conditional generation and efficient in terms of requiring less training data and computational resources. We further investigate the potential applications for our spatial control method through axial super-resolution for medical images. Our code is available at \url{https://github.com/Ahn-Ssu/VCM}
BibTeX
@inproceedings{jinah2025volumetric,
  author = {Park Jinah and Suhyun Ahn and Wonjung Park and Jihoon Cho and Jinah Park},
  title = {{Volumetric Conditioning Module to Control Pretrained Diffusion Models for 3D Medical Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00019},
}
Figure