StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis

Minguk Kang, Joonghyuk Shin, Jaesik Park

Introduction

The field of Generative Adversarial Networks (GANs) has experienced rapid growth, yet it remains plagued by a reproducibility crisis. Discrepancies in deep learning frameworks, inconsistent data preprocessing, and varying evaluation protocols make fair comparisons between models nearly impossible. StudioGAN addresses these challenges by providing a comprehensive open-source library and benchmark designed to standardize GAN research and move beyond the "Wild West" of inconsistent reporting.

The StudioGAN Framework and Taxonomy

StudioGAN introduces a modular PyTorch-based library that categorizes GAN components into five fundamental pillars: Architecture, Conditioning Goal, Adversarial Loss, Regularization, and Data-Efficiency techniques. This taxonomy allows researchers to isolate and study the impact of specific components on overall performance, ensuring that improvements are attributed to the correct innovations rather than hidden implementation details.

Taxonomy of GAN components categorized into five pillars: Architecture, Conditioning goal, Adversarial loss, Regularization, and Data-efficient techniques.

Standardizing the Evaluation Pipeline

A critical contribution of StudioGAN is the identification of "hidden" variables in the evaluation pipeline. The authors demonstrate that seemingly minor choices, such as the type of image resizer or the handling of quantization (8-bit conversion), can significantly alter metrics like the Fréchet Inception Distance (FID). This "resizer trap" can lead to artificial performance gains if a model's output happens to align with the aliasing artifacts of a specific resizer.

The StudioGAN training and evaluation pipeline, highlighting the impact of pre-processing, quantization, and post-processing on metric calculation.

To mitigate these issues, StudioGAN establishes a strict protocol using "backbone-friendly" anti-aliasing resizers (e.g., PIL.BICUBIC) and consistent quantization. Furthermore, the benchmark moves beyond the aging InceptionV3 backbone, incorporating modern self-supervised backbones like SwAV and vision transformers like Swin-T to provide a more robust assessment of image quality and diversity.

Benchmarking and Key Insights

The authors evaluated over 30 GAN variants across diverse datasets, including industry staples like CIFAR10 and ImageNet, as well as high-resolution sets like AFHQv2 and FFHQ. They also introduced "Baby/Papa/Grandpa-ImageNet" subsets to explore the relationship between classification difficulty and generation quality.

Key findings from this extensive benchmark include:

  • The Resizer Trap: The choice of resizer can significantly swing FID scores. The authors recommend matching the resizer to the specific evaluation backbone used to ensure fairness.
  • Architecture Strengths: While StyleGAN is the state-of-the-art for low-variance datasets like human faces, BigGAN and its variants often maintain better structural coherency on complex, multi-class datasets like ImageNet.
  • The FID Mirage: A low FID score does not always correlate with visual quality or diversity. Models can "hack" the FID by aligning with the InceptionV3 feature space while lacking actual diversity, making multi-metric evaluation (e.g., Precision/Recall, Density/Coverage) essential.
  • Inference Efficiency: Despite the high quality of modern diffusion models, GANs remain vastly superior in speed, often generating images 1,000x faster (e.g., 0.04s for StyleGAN-XL vs. 50s for ADM-G).

Conclusion

StudioGAN provides the necessary infrastructure to move GAN research toward a more rigorous and reproducible future. By standardizing preprocessing and diversifying evaluation metrics, it reveals the true state of the art and sets a new bar for transparency in generative AI. Future work will focus on scaling GANs for complex text-to-image synthesis and bridging the gap between automated metrics and human perception of realism.

Summary updated

BibTeX
@article{kang2023studiogan,
  author = {Minguk Kang and Joonghyuk Shin and Jaesik Park},
  title = {{StudioGAN: A Taxonomy and Benchmark of GANs for Image Synthesis}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3306436},
}
Figure