Abstract: Synthetic images generated from deep generative models have the potential to address data scarcity and data privacy issues. The selection of synthesis models is mostly based on image quality measurements, and most researchers favor synthetic images that produce realistic images, i.e., images with good fidelity scores, such as low Fr'echet Inception Distance (FID) and high Peak Signal-To-Noise Ratio (PSNR). However, the quality of synthetic images is not limited to fidelity, and a wide spectrum of metrics should be evaluated to comprehensively measure the quality of synthetic images. In addition, quality metrics are not truthful predictors of the utility of synthetic images, and the relations between these evaluation metrics are not yet clear. In this work, we have established a comprehensive set of evaluators for synthetic images, including fidelity, variety, privacy, and utility. By analyzing more than 100k chest X-ray images and their synthetic copies, we have demonstrated that there is an inevitable trade-off between synthetic image fidelity, variety, and privacy. In addition, we have empirically demonstrated that the utility score does not require images with both high fidelity and high variety. For intra- and cross-task data augmentation, mode-collapsed images and low-fidelity images can still demonstrate high utility. Finally, our experiments have also showed that it is possible to produce images with both high utility and privacy, which can provide a strong rationale for the use of deep generative models in privacy-preserving applications. Our study can shore up comprehensive guidance for the evaluation of synthetic images and elicit further developments for utility-aware deep generative models in medical image synthesis.
Introduction
In medical imaging, the development of robust AI models is often hindered by data scarcity, particularly for rare diseases, and stringent data privacy regulations like HIPAA and GDPR. Synthetic data generated by deep generative models offers a promising solution. However, current evaluation methods often rely on "entangled" metrics like the Fréchet Inception Distance (FID), which conflate image realism (fidelity) with dataset diversity (variety). This ambiguity can lead to misleading conclusions; for instance, a model might achieve a low FID by memorizing training data, posing a significant privacy risk, or it might be dismissed for low fidelity despite providing high utility for downstream tasks.
A Four-Dimensional Evaluation Framework
To address these limitations, we propose a comprehensive framework that disentangles the quality of synthetic images into four distinct dimensions:
- Fidelity ($F$): The visual realism of individual images.
- Variety ($V$): The diversity of the generated dataset, measured using an information-based proxy (JPEG compression of the mean image).
- Privacy ($P$): The resistance to membership inference attacks, ensuring the model does not leak sensitive training data.
- Utility ($U$): The practical benefit of the synthetic data for downstream clinical tasks, categorized as intra-task (same disease) or cross-task (transfer learning).
Key Insights and Trade-offs
Our analysis of over 100,000 chest X-ray images reveals a fundamental trade-off between fidelity, variety, and privacy. We observed a strong negative correlation (Pearson -0.92) between fidelity and variety: models that produce highly realistic images often struggle with diversity.
Crucially, we found that "perfect" images are not a prerequisite for high utility. In intra-task scenarios, such as detecting pleural effusion, "mode-collapsed" images—which lack variety but capture essential pathological features—significantly improved classification accuracy. For cross-task utility, Latent Diffusion Models (LDM) outperformed high-fidelity models like StyleGAN2. Despite producing images that radiologists identified as "fake" due to texture issues, LDMs' superior variety provided more robust features for transfer learning to different clinical domains, such as pediatric pneumonia detection.
Validation and Metric Alignment
To ensure our metrics reflect clinical reality, we validated them against the perceptions of expert radiologists. Our proposed measures for fidelity and variety showed significant alignment with human ratings ($p < 0.05$), whereas traditional FID failed to provide meaningful insights in the presence of mode collapse.
Conclusion
This study demonstrates that the pursuit of visual "perfection" in synthetic medical imaging may be counterproductive. By understanding the trade-offs between fidelity, variety, and privacy, researchers can develop "utility-aware" generative models tailored to specific clinical needs. Our findings provide a strong rationale for using synthetic data in privacy-sensitive applications, showing that high utility and privacy can indeed coexist. Future work will explore these principles in 3D modalities and under formal differential privacy constraints.
Summary updated