Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generation

Lorenzo Tronchin, Tommy Löfstedt, Paolo Soda, Valerio GuarrasiarXiv 2025 · DOI · arXiv

TL;DR

Generative Adversarial Networks (GANs) often struggle to capture the full diversity of medical images, risking missing rare pathologies due to mode collapse. This paper introduces a multi-objective optimization method to select an optimal ensemble of GANs—combining different architectures and training epochs—to maximize image fidelity and diversity while minimizing computational redundancy.

What Problem Does This Paper Address?

Medical imaging AI requires synthetic data that is both high-fidelity (looks realistic) and diverse (covers rare diseases and patient variations). While GANs generate high-quality images quickly, they suffer from the "generative learning trilemma," frequently falling victim to mode collapse or failing to cover the entire real data distribution.

Selecting a single "best" GAN is difficult and inherently limits diversity, while a "naive" ensemble of all available GANs introduces low-quality images and massive computational overhead. The authors tackle the challenge of systematically building a lean, high-performing GAN ensemble specifically optimized for medical data generation.

Methods

The authors propose a Pareto-based multi-objective optimization framework to select the best subset of GANs from a pool of 110 unique configurations (22 distinct architectures sampled at 5 different training epochs).

  • Fidelity & Diversity Metrics: They use a combination of density (how closely generated samples pack around real data cores) and coverage (how well the generated samples span the entire real data manifold).
  • Intra-d vs. Inter-d Optimization: The framework simultaneously maximizes Intra-d (similarity between the ensemble's data and real data) and minimizes Inter-d (similarity between different GANs in the ensemble to eliminate redundancy).
  • Self-Supervised Backbone: Instead of standard ImageNet-pretrained classifiers, they leverage SwAV (Swapping Assignments between Views), a self-supervised model, to extract robust image embeddings for evaluation.
Figure 1: An illustration of the manifold occupied by real data, R, alongside the manifold covered by various GAN configurations, Si. Single: Showcases three scenarios: Si entirely within the R, indicating high fidelity; Si partially overlapping with the R representing medium fidelity; Si entirely outside the R suggesting low fidelity. Single GAN selection inherently has a low diversity due to limited real data coverage. Naive (A): Depicts the collective space covered by all GANs, encompassing areas both inside and outside the R. This approach results in medium fidelity and high diversity. G∗: Demonstrates the space covered by an ensemble of GANs selected through the proposed optimisation method, which aims to maximise coverage of the R to ensure diversity discharging GAN outside the R, to ensure fidelity, using as few GANs as possible.

Datasets

The method was evaluated across three distinct medical image datasets with varying modalities:

  1. PneumoniaMNIST: 5,856 pediatric chest X-ray images for binary classification (Normal vs. Pneumonia).
  2. BreastMNIST: 780 breast ultrasound images simplified into binary classification (Normal/Benign vs. Malignant).
  3. AIforCOVID: 820 chest X-ray scans from Italian hospitals predicting clinical severity (Mild vs. Severe).

Key Insights and Ideas

  • Epoch-level Diversity: The min-max game dynamics of GANs continuously shift the generator network during training. Sampling the same GAN at different training iterations yields models that capture entirely different data modes, providing a "free" source of ensemble diversity without extra training overhead.
  • The Backbone Bias: Standard ImageNet-pretrained models introduce bias when evaluating medical images. Surprisingly, fine-tuning these backbones on medical data led to overfitting and worse ensemble selection. Self-supervised backbones (SwAV) proved to be the most robust and generalizable.
  • Diversity Trumps Fidelity for Downstream Tasks: For training downstream diagnostic models, data diversity (coverage) is more critical than flawless fidelity (density). Once a diversity threshold is met, downstream performance stabilizes, whereas high fidelity alone does not guarantee downstream utility.

Key Results

The optimized ensemble ($G^*$) consistently outperformed single GANs and naive combinations when training a downstream ResNet18 classifier from scratch:

  • On PneumoniaMNIST, the synthetic data from $G^*$ actually outperformed the real training data baseline, achieving a positive performance gap ($\gamma_{RS}$) of +5.5% (and up to +7.2% in the Oracle setup).
  • On BreastMNIST and AIforCOVID, $G^*$ significantly closed the performance gap compared to average single models, reducing the negative gap from -50.2% and -44.2% down to just -7.6% and -5.6%, respectively.
  • $G^*$ achieved these results while using only a fraction of the total models (30–38 GANs out of 110), reducing the GPU footprint by roughly 3x compared to the naive all-inclusive approach.

Limitations and Future Work

  • Static Contributions: Once the ensemble is selected, the relative contribution of each GAN is fixed during downstream training. The authors suggest moving toward dynamic ensembles that adaptively weight models based on feedback from the downstream task.
  • Upfront Computational Cost: The current method requires training a large pool of networks (22 architectures) before selection can occur. Future work will focus on optimizing the ensemble search within a strict, predefined computational budget encompassing training, search, and downstream tasks.

Summary updated

Abstract

The advancement of generative AI, particularly in medical imaging, confronts the trilemma of ensuring high fidelity, diversity, and efficiency in synthetic data generation. While Generative Adversarial Networks (GANs) have shown promise across various applications, they still face challenges like mode collapse and insufficient coverage of real data distributions. This work explores the use of GAN ensembles to overcome these limitations, specifically in the context of medical imaging. By solving a multi-objective optimisation problem that balances fidelity and diversity, we propose a method for selecting an optimal ensemble of GANs tailored for medical data. The selected ensemble is capable of generating diverse synthetic medical images that are representative of true data distributions and computationally efficient. Each model in the ensemble brings a unique contribution, ensuring minimal redundancy. We conducted a comprehensive evaluation using three distinct medical datasets, testing 22 different GAN architectures with various loss functions and regularisation techniques. By sampling models at different training epochs, we crafted 110 unique configurations. The results highlight the capability of GAN ensembles to enhance the quality and utility of synthetic medical images, thereby improving the efficacy of downstream tasks such as diagnostic modelling.
BibTeX
@misc{tronchin2025beyond,
  author = {Lorenzo Tronchin and Tommy Löfstedt and Paolo Soda and Valerio Guarrasi},
  title = {{Beyond a Single Mode: GAN Ensembles for Diverse Medical Data Generation}},
  year = {2025},
  eprint = {2503.24258},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.24258},
}
Figure