Paired Diffusion: Generation of related, synthetic PET-CT-Segmentation scans using Linked Denoising Diffusion Probabilistic Models

Rowan Bradbury, Katherine A. Vallis, Bartlomiej W. Papiez

Abstract

Deep learning models in medical imaging are often bottlenecked by the scarcity of high-quality, annotated multi-modal data. This paper introduces Paired Diffusion, a generative framework that simultaneously creates synthetic CT and PET scans alongside their corresponding ground-truth segmentation masks. By utilizing linked Denoising Diffusion Probabilistic Models (DDPMs) and a novel "noise-seeding" strategy, the framework generates massive, fully-annotated datasets, effectively bypassing the traditional costs and privacy constraints associated with manual medical image labeling.

Introduction

Radiotherapy planning and other complex medical tasks require vast amounts of annotated data. However, acquiring such data is difficult due to HIPAA/GDPR privacy concerns and the extreme cost of expert manual annotation. While standard generative models like GANs can create realistic images, they often fail to generate the associated labels (segmentation masks) or maintain spatial consistency across different modalities (e.g., ensuring a tumor appears in the same location on both a synthetic PET and CT scan). Paired Diffusion addresses these challenges by ensuring structural alignment across multiple generated channels.

Methodology

The authors propose a multi-network architecture where each modality (CT, PET, and Segmentation Mask) is handled by a dedicated DDPM. These networks are synchronized through several key mechanisms:

  • Conditional Encoders & CBAM: Sub-networks use conditional encoders to share features. A Convolutional Block Attention Module (CBAM) dynamically adjusts the influence of these features based on the diffusion timestep.
  • Timestep-Controlled Low-Pass (TLP) Filter: To clean conditional inputs, the authors developed a learnable frequency-space filter. Unlike previous complex filters, the TLP filter uses a simpler two-variable approach to remove noise while preserving semantic structures like organ shapes.
The proposed TLP filter compared against the existing FF-Parser architecture.
  • Noise Seeding: Rather than starting from pure noise, the process begins by adding 80-90% noise to a real image. This "seeds" the generation with a realistic global structure while allowing the model to produce unique, synthetic details.
  • Modified Perceptual Loss: A VGG-16 based perceptual loss forces the networks to align high-level features, preventing artifacts like "bright" bone in CT from being misinterpreted as tumor signals in the PET channel.

Key Results

The study utilized the HaN-Seg and HECKTOR 2022 datasets for evaluation. Key findings include:

  • Improved Convergence: Downstream segmentation models (e.g., TransUNet) trained on synthetic data converged significantly faster.
  • Accuracy Gains: Pre-training on synthetic paired data led to a statistically significant improvement in segmentation accuracy ($p = 0.00046$) in data-constrained scenarios.
  • Qualitative Success: The "Seeded" approach produced synthetic images that maintained perfect spatial alignment between CT, PET, and Mask channels, a major hurdle for previous DDPM implementations.
Example of aligned multi-modal generation showing CT and PET consistency.

Discussion and Future Work

While Paired Diffusion shows great promise, the authors noted that perceptual loss must be carefully tuned to avoid misaligning unrelated bright features. Additionally, training multiple DDPMs simultaneously is computationally expensive. Future research will focus on scaling the model to full 3D consistency for clinical radiotherapy applications and exploring faster sampling methods like GANs or VAEs to provide the initial structural "seed."

Summary updated

BibTeX
@inproceedings{bradbury2024paired,
  author = {Rowan Bradbury and Katherine A. Vallis and Bartlomiej W. Papiez},
  title = {{Paired Diffusion: Generation of related, synthetic PET-CT-Segmentation scans using Linked Denoising Diffusion Probabilistic Models}},
  booktitle = {ISBI},
  year = {2024},
  doi = {10.1109/isbi56570.2024.10635593},
}
Figure