TL;DR
This paper introduces FetalFlex, a novel diffusion model for generating realistic fetal ultrasound (US) images with precise anatomical control. It addresses the challenge of limited and imbalanced medical data by enabling the synthesis of both normal and abnormal US images, crucial for training robust AI models and improving prenatal diagnostics.
What Problem Does This Paper Address?
Fetal ultrasound (US) examinations are vital for monitoring fetal development and screening for anomalies. However, acquiring comprehensive, multi-plane US images is challenging due to fetal movement and the scarcity of annotated data, especially for rare anomalies. This data limitation hinders the training of AI models for accurate diagnosis and anomaly detection. Existing generative models often struggle with the complexity of fetal anatomy, lack fine-grained control, and are primarily focused on normal cases, leaving a significant gap in generating realistic abnormal fetal US images for robust AI development.
Methods
FetalFlex leverages a Latent Diffusion Model (LDM) as its backbone, enhanced with ControlNet for controllability. Key innovations include:
- Anatomical Structure Guidance: The model uses anatomical structures extracted from images (via object detection) as control conditions, alongside layout maps and global text prompts.
- Pre-alignment Module: This module aligns sparse layout map features with image features, improving the model's attention to anatomical relationships.
- Conditional Inpainting (Repaint Strategy): Adapted for US data, this technique uses a Region of Interest (ROI) mask to refine generated images, ensuring consistency within specific areas.
- Two-Stage Sampling Strategy:
- Scale-Space Adaptive (SSA) Sampling: This stage uses Anisotropic Diffusion (AD) to enhance tissue boundaries and adapt to the unique characteristics of US imaging.
- Organ-Scale Mask (OSM) Sampling: This stage allows for fine-grained editing and refinement, particularly useful for simulating abnormalities.
- Ensemble Loss Function: Combines Mean Squared Error (MSE) for local ROI fidelity and Structural Similarity Index Measure (SSIM) for global image quality.
- BERT for Text Encoding: Replaces CLIP with a BERT model pre-trained on medical abstracts to better capture medical semantic information.
Datasets
- Training and Evaluation: A multi-center dataset of 3,600 fetal US images across three planes (thalamic transverse, facial sagittal, upper abdominal transverse) from women aged 20-24 weeks gestation.
- Abnormal Samples: 360 real abnormal samples were collected for specific conditions (hydrocephalus, cleft lip/palate, duodenal atresia/stenosis) to construct OOD datasets.
- Downstream Tasks: Datasets were augmented with synthetic data generated by FetalFlex for classification and anomaly detection experiments.
Key Insights and Ideas
- Anatomical Control is Key: Precisely controlling anatomical structures, not just general image features, is crucial for generating realistic and clinically relevant fetal US images, especially for anomalies.
- Diffusion Models for Medical Synthesis: Diffusion models, with their inherent flexibility and controllability, are well-suited for medical image synthesis, particularly for complex anatomical structures.
- Bridging ID and OOD: FetalFlex's ability to generate both in-distribution (normal) and out-of-distribution (abnormal) images without requiring abnormal training data is a significant advancement.
- Two-Stage Sampling for Refinement: The coarse-to-fine sampling strategy effectively balances global consistency with local detail, crucial for high-fidelity image generation.
- Synthetic Data for Downstream Tasks: Generated synthetic data can significantly improve the performance of deep learning models in classification and anomaly detection, even outperforming models trained solely on real data in some scenarios.
Key Results
- Quantitative Performance: FetalFlex achieved state-of-the-art results across multiple image quality metrics (PSNR, MS-SSIM, FID, LPIPS) compared to baseline methods like Stable Diffusion, ControlNet, and Repaint. For example, it achieved the highest PSNR (23.686) and MS-SSIM (0.876), and the lowest FID (51.103) and LPIPS (0.144).
- Qualitative Results: Generated images were visually superior, exhibiting finer details, better contrast, and fewer artifacts, closely matching expert visual assessments.
- Downstream Task Improvement: Synthetic images from FetalFlex significantly improved the performance of six typical deep models in downstream classification and anomaly detection tasks. For instance, models trained on a mixed 50% real + 50% generated dataset achieved the best results across classification metrics.
- Anomaly Synthesis: FetalFlex successfully simulated various abnormalities (structural displacement, size changes, absence of structures) with high fidelity, closely resembling real abnormal cases.
Limitations and Future Work
- Data Heterogeneity: The authors acknowledge that the collected fetal US dataset was not manually cleaned, leading to some low-quality images and high data heterogeneity, which could be improved in future work.
- Standardized Metrics: Currently, no standardized metrics exist to quantify controllable generation capabilities in fetal US imaging. Validation relied on visualization and downstream task performance.
- Future Work: The authors aim to develop a generalized framework for fetal US synthesis by incorporating the semantic capabilities of large language models into diffusion models. They also suggest further research into data cleaning and the development of more robust evaluation metrics.
Summary updated