TL;DR
This paper introduces a novel method for generating realistic 3D medical images with synthetic tumors, conditioned on radiomics features. This approach addresses the challenge of limited medical data by enabling precise control over tumor characteristics like size, shape, and texture, paving the way for improved training of diagnostic models and personalized treatment planning.
What Problem Does This Paper Address?
Obtaining large, diverse datasets for medical image analysis, especially for 3D modalities like CT and MRI, is a significant hurdle due to privacy concerns and the rarity of certain conditions. Existing generative models, particularly GANs, often struggle with generating diverse and accurate 3D medical images, suffering from issues like mode collapse. This limits their utility in clinical settings where understanding subtle tumor variations is crucial for diagnosis, prognosis, and treatment planning. Radiomics features, which capture quantitative information about tumor characteristics, offer a promising avenue for controlling generative models, but their direct application for synthesizing complex 3D tumors with precise control remains an open challenge.
Methods
The proposed method employs a two-stage approach for tumor synthesis:
- Tumor Shape Generator: This component uses a conditional GAN (specifically, an adaptation of GigaGAN) that leverages radiomics features related to shape. It's designed to generate a 3D tumor mask. The use of attention mechanisms (self-attention and cross-attention) is key here, allowing the model to capture long-range relationships essential for defining tumor shape.
- Tumor Texture Generator: This component utilizes a Brownian Bridge Diffusion Model (BBDM), a type of diffusion model adapted for image-to-image translation. It takes the masked image (with the generated tumor mask) and radiomics features related to texture as input to generate the tumor's texture. To handle the high dimensionality of 3D volumes, the model employs VQ-GAN for image compression into a latent space before applying the diffusion process. This allows for flexible manipulation, including tumor synthesis and removal, by switching between self-attention (for removal) and cross-attention (for synthesis with texture conditioning).
Datasets
The models were trained and evaluated on datasets from four different organs across two modalities:
- Kidney: CT data from the KiTS23 dataset.
- Lung: CT data from the NSCLS dataset.
- Breast: MRI data from a private dataset.
- Brain: MRI data from the BraTS2021 dataset.
The datasets were split with 80% for training and 20% for testing.
Key Insights and Ideas
- Radiomics as a Control Knob: The core idea of using radiomics features as explicit conditions for tumor generation is powerful. It moves beyond simple image-to-image translation to a more interpretable and controllable synthesis process.
- Separating Shape and Texture Generation: Decomposing the problem into generating shape first and then texture allows for more targeted control and potentially better results, leveraging different model architectures suited for each task.
- Diffusion Models for Texture Fidelity: The use of BBDM for texture generation demonstrates its effectiveness in capturing subtle textural details, outperforming GAN-based approaches in this regard.
- Controllable Tumor Manipulation: The ability to not only synthesize tumors but also remove them and alter their shape and texture based on user-defined radiomics features offers significant clinical potential.
- Expert Validation: The fact that even board-certified radiologists struggled to distinguish synthesized images from real ones is a strong indicator of the model's realism.
Key Results
- Quantitative Performance: The BBDM-based texture generator achieved the best performance in quantitative evaluations (PSNR and SSIM) across all organs and modalities compared to GAN and LDM baselines. For instance, in tumor synthesis for the kidney, the proposed method achieved a PSNR of 33.95 and SSIM of 0.9616.
- Radiomics Feature Replication: The model demonstrated high fidelity in replicating both texture and shape features, with correlation coefficients (Pearson and Spearman) often exceeding 0.8 for MRI images, particularly for the brain.
- Downstream Task Improvement: Training a segmentation model (nnU-Net) with the synthesized images led to a performance improvement (measured by DICE scores), especially in the less data-abundant breast tumor segmentation task.
- Qualitative Results: Visual results showed that the proposed method could generate realistic tumors with controlled shapes and textures. Expert evaluations indicated that synthesized images were highly indistinguishable from real ones, with an average accuracy of 57.5% in distinguishing them.
Limitations and Future Work
- Surrounding Tissue: The current model does not explicitly account for surrounding tissue changes like edema or tumor-induced distortion, which can be important in real-world scenarios.
- Generalization: The reliance on radiomics features extracted from a single dataset for a given organ and modality might limit the model's generalization to unseen data distributions.
- Future Work: The authors suggest that the ability to generate tumors with desired shapes, locations, and textures could enable various simulations crucial for personalized treatment plans, disease progression modeling, and directing patients to alternative treatments. Further work could explore incorporating surrounding tissue information and improving generalization.
Summary updated