You Don't Have to Be Perfect to Be Amazing: Unveil the Utility of Synthetic Images

Xiaodan Xing, Federico Felder, Yang Nan, Giorgos Papanastasiou, Simon Walsh, Guang Yang

TL;DR

Synthetic medical images don't need to be "perfect" to be incredibly useful. This paper introduces a four-dimensional framework—fidelity, variety, privacy, and utility—to show that even "flawed" images (like those suffering from mode collapse or low visual realism) can significantly boost the performance of clinical AI models while protecting patient privacy.

What Problem Does This Paper Address?

In medical imaging, researchers often turn to synthetic data to solve two massive hurdles: data scarcity (rare diseases) and data privacy (strict HIPAA/GDPR regulations). However, we currently lack a clear way to measure if a synthetic dataset is actually "good."

Most researchers rely on "entangled" metrics like the Fréchet Inception Distance (FID). The problem? FID lumps together image quality (fidelity) and image diversity (variety). A dataset could have a "good" FID score but actually be a set of near-identical copies of real patients—a massive privacy breach. Conversely, a model might produce "ugly" images that are actually excellent for training a classifier. The authors set out to disentangle these metrics and understand the "utility" of synthetic images: how much they actually help a downstream AI task.

Methods

The authors propose a comprehensive four-dimensional framework for evaluation:

  1. Fidelity ($F$): How realistic is the image?
  2. Variety ($V$): How diverse is the dataset? (Measured using the JPEG file size of the mean image—a clever proxy for "blurriness" vs. "sharpness").
  3. Privacy ($P$): Does the model just memorize and "leak" training data? They used a membership inference attack based on Hamming distances in a quantized latent space (via VQ-VAE).
  4. Utility ($U$): Does adding this data actually improve a real-world model?
  5. Intra-task utility: Adding synthetic images to the training set to improve accuracy on the same task (e.g., detecting pleural effusion).
  6. Cross-task utility: Using a model trained on synthetic data as a "feature extractor" for a completely different task (e.g., detecting pediatric pneumonia).

They tested two state-of-the-art generative architectures: StyleGAN2 (known for high fidelity) and Latent Diffusion Models (LDM) (known for diversity).

Datasets

The study utilized a massive scale of data:

  • CheXpert: Over 100,000 chest X-ray images. They focused specifically on detecting Pleural Effusion (PE).
  • Pediatric X-ray Dataset: A separate dataset of 5,863 images (pneumonia vs. normal) used specifically to test cross-task utility.

Key Insights and Ideas

  • The Fidelity-Variety Trade-off: There is a strong negative correlation (Pearson -0.92) between how realistic images look and how diverse they are. You can have one, but rarely both at peak levels.
  • Mode Collapse as a "Feature," Not a Bug: In a surprising twist, the authors found that "mode-collapsed" images (where the generator produces very similar-looking images) actually helped intra-task classification. Why? Because the generator essentially "distilled" the most important visual features of a pathology (like the specific opacity of fluid in the lungs), making it easier for the classifier to learn.
  • LDM: The Cross-Task Champion: While radiologists easily identified LDM-generated images as "fake" due to poor texture (low fidelity), these images were superior for cross-task utility. Their high variety meant the features learned were more robust and generalized better to entirely different clinical problems.
  • Privacy and Utility Can Coexist: The results suggest that for transfer learning (cross-task), you can achieve high utility without sacrificing privacy, providing a strong rationale for using synthetic data in privacy-sensitive medical applications.

Key Results

  • Human vs. Metric: The proposed metrics for fidelity and variety matched human (radiologist) perception perfectly ($p < 0.05$), whereas traditional FID failed to provide a valid analysis when mode collapse occurred.
  • Performance Boost: Adding synthetic data (even from mode-collapsed StyleGAN2 models) significantly improved classification accuracy for Pleural Effusion compared to using real data alone or simple augmentations like flipping and rotating.
  • Generalization: Models trained on high-variety LDM data provided "powerful features" that facilitated downstream tasks on the pediatric pneumonia dataset, even though the LDM images lacked fine-grained realistic textures.

Limitations and Future Work

The authors acknowledge a few key limitations:

  • Differential Privacy: While they measured privacy via membership inference attacks, they did not implement formal Differential Privacy (DP) during training, which usually incurs a significant "utility cost."
  • Domain Specificity: The study is focused on 2D X-rays. It remains to be seen if these insights hold for 3D modalities like MRI or CT, where the "curse of dimensionality" makes both synthesis and evaluation much harder.
  • Utility-Aware Models: The paper concludes that we should stop chasing "perfect-looking" images and instead focus on developing "utility-aware" generative models that prioritize the specific needs of the downstream clinical task.

Summary updated

Abstract

Synthetic images generated from deep generative models have the potential to address data scarcity and data privacy issues. The selection of synthesis models is mostly based on image quality measurements, and most researchers favor synthetic images that produce realistic images, i.e., images with good fidelity scores, such as low Fr\'echet Inception Distance (FID) and high Peak Signal-To-Noise Ratio (PSNR). However, the quality of synthetic images is not limited to fidelity, and a wide spectrum of metrics should be evaluated to comprehensively measure the quality of synthetic images. In addition, quality metrics are not truthful predictors of the utility of synthetic images, and the relations between these evaluation metrics are not yet clear. In this work, we have established a comprehensive set of evaluators for synthetic images, including fidelity, variety, privacy, and utility. By analyzing more than 100k chest X-ray images and their synthetic copies, we have demonstrated that there is an inevitable trade-off between synthetic image fidelity, variety, and privacy. In addition, we have empirically demonstrated that the utility score does not require images with both high fidelity and high variety. For intra- and cross-task data augmentation, mode-collapsed images and low-fidelity images can still demonstrate high utility. Finally, our experiments have also showed that it is possible to produce images with both high utility and privacy, which can provide a strong rationale for the use of deep generative models in privacy-preserving applications. Our study can shore up comprehensive guidance for the evaluation of synthetic images and elicit further developments for utility-aware deep generative models in medical image synthesis.
BibTeX
@inproceedings{xing2023you,
  author = {Xiaodan Xing and Federico Felder and Yang Nan and Giorgos Papanastasiou and Simon Walsh and Guang Yang},
  title = {{You Don't Have to Be Perfect to Be Amazing: Unveil the Utility of Synthetic Images}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43904-9\_2},
}

Loading related content...

Figure