Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation

Xinrong Hu, Yu-Jen Chen, Tsung-Yi Ho, Yiyu Shi

Introduction

In medical imaging, segmentation—the precise delineation of anatomical structures or pathological regions—is a cornerstone of diagnosis and treatment planning. However, training deep learning models for this task typically requires large datasets with pixel-perfect "ground truth" labels. Generating these labels is a significant bottleneck, as it demands hours of meticulous work from highly trained radiologists.

Weakly Supervised Semantic Segmentation (WSSS) offers a solution by training models on significantly cheaper labels, such as image-level tags (e.g., "tumor present" vs. "no tumor"). While traditional WSSS methods rely on Class Activation Maps (CAMs), these often produce blurry and imprecise results. This paper introduces a novel framework using Conditional Diffusion Models (CDM) to generate high-quality segmentation masks from image-level labels with unprecedented speed and accuracy.

Methodology

The Gradient Trick and Finite Difference Approximation

The core innovation of this research lies in how it leverages the generative power of diffusion models. Rather than performing a full denoising process to reconstruct an image—a computationally expensive task—the authors focus on the model's internal "noise prediction."

By training a standard CDM using a U-Net architecture, the model learns to predict the noise added to an image, conditioned on a binary class label. To identify the target object, the researchers calculate the sensitivity of the predicted noise to the input condition. Since class labels are discrete, a direct derivative cannot be taken. Instead, the authors utilize a Finite Difference Approximation: they perturb the condition embedding by a small value ($\tau$) and measure the resulting change in predicted noise. This "gradient map" effectively highlights the pixels most sensitive to the presence of the target class, forming the basis of the segmentation mask.

Efficiency and Classifier Guidance

Previous diffusion-based segmentation methods were notoriously slow, often requiring over 400 iterations to produce a single mask. The proposed method achieves high-quality results in approximately 10 iterations ($R=10$), representing a nearly 40-fold increase in inference speed. Additionally, the authors introduce Classifier Guidance (CG-CDM), which employs an external classifier to further refine the diffusion model's output, resulting in sharper and more anatomically consistent masks.

Experimental Results

The framework was evaluated on two primary medical benchmarks: the BraTS (Brain Tumor Segmentation) dataset and the CHAOS (Combined Healthy Abdominal Organ Segmentation) dataset.

Performance Metrics

On the BraTS dataset, the CG-CDM method established a new state-of-the-art for weakly supervised methods:

  • Accuracy: It achieved a Dice score of 0.563, a dramatic improvement over traditional CAM-based methods (which typically range between 0.23 and 0.30) and the previous diffusion-based state-of-the-art (0.456).
  • Inference Speed: The time required per image was reduced from 116 seconds (using previous diffusion methods) to just 2.78 seconds.

Qualitative analysis confirmed that the masks produced by this method are significantly tighter and more anatomically accurate than the coarse "blobs" generated by traditional weakly supervised approaches.

Discussion and Key Insights

Noise as a Semantic Map

A key insight of this work is that the noise predicted by a diffusion model is not merely random; it is deeply encoded with the semantic content of the image. By analyzing how this noise shifts in response to label changes, the model reveals high-resolution structural information. Furthermore, by focusing on the gradient of the noise rather than a full image reconstruction, the method avoids the "background distortion" or hallucinations that plagued earlier diffusion approaches.

The Role of Noise Levels

The researchers identified a "sweet spot" for the initial noise level ($Q$). If the image is too clean, the model lacks the sensitivity to show semantic variations; if it is too noisy, structural information is lost. An optimal starting point of $Q=400$ (out of 1000) was found to provide the best balance for medical segmentation tasks.

Conclusion

This paper demonstrates that Conditional Diffusion Models can be transformed into efficient, precise tools for weakly supervised medical image segmentation. By utilizing the sensitivity of noise predictions, the proposed CG-CDM framework bridges the gap between the high cost of expert annotation and the need for accurate diagnostic AI. While a performance gap still exists compared to fully supervised models (which can reach Dice scores of ~0.90), this work represents a significant leap forward in the feasibility of using weakly labeled data for clinical applications.

Summary updated

Abstract

Recent advances in denoising diffusion probabilistic models have shown great success in image synthesis tasks. While there are already works exploring the potential of this powerful tool in image semantic segmentation, its application in weakly supervised semantic segmentation (WSSS) remains relatively under-explored. Observing that conditional diffusion models (CDM) is capable of generating images subject to specific distributions, in this work, we utilize category-aware semantic information underlied in CDM to get the prediction mask of the target object with only image-level annotations. More specifically, we locate the desired class by approximating the derivative of the output of CDM w.r.t the input condition. Our method is different from previous diffusion model methods with guidance from an external classifier, which accumulates noises in the background during the reconstruction process. Our method outperforms state-of-the-art CAM and diffusion model methods on two public medical image segmentation datasets, which demonstrates that CDM is a promising tool in WSSS. Also, experiment shows our method is more time-efficient than existing diffusion model methods, making it practical for wider applications. The codes are available at https://github.com/xhu248/cond_ddpm_wsss .
BibTeX
@inproceedings{hu2023conditional,
  author = {Xinrong Hu and Yu-Jen Chen and Tsung-Yi Ho and Yiyu Shi},
  title = {{Conditional Diffusion Models for Weakly Supervised Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=oRZGXirsV9t},
}
Figure