A Classification-Based Study of Covariate Shift in GAN Distributions

Shibani Santurkar, Ludwig Schmidt, Aleksander MadryICML 2018 · OpenReview

Generative Adversarial Networks (GANs) have achieved remarkable success in generating highly realistic images. However, a fundamental question persists: do GANs capture the entire distribution of the training data, or do they merely memorize a few "easy" modes? Existing evaluation metrics, such as the Inception Score or Fréchet Inception Distance (FID), primarily assess visual quality and offer only a partial view of distributional diversity. Other methods for detecting "mode collapse" often rely on subjective human inspection or simplified toy models.

This paper introduces a scalable, quantitative framework to evaluate the diversity of GAN distributions by framing diversity loss as a form of covariate shift. The authors demonstrate that even state-of-the-art GANs suffer from significant mode collapse and boundary distortion, effectively capturing a distribution equivalent to only a few hundred real training samples.

Methodology: Measuring Covariate Shift

The authors propose using classification performance as a proxy to detect two specific types of covariate shift in GANs:

  1. Mode Collapse: To quantify this, a GAN is trained on a balanced dataset (e.g., equal numbers of "smiling" and "not smiling" faces). A separate classifier, trained on real data, acts as an "annotator" to label a large batch of GAN-generated images. If the GAN has maintained the original balance, the labels should be evenly distributed; skewed results indicate collapse onto specific attributes.
  2. Boundary Distortion: This occurs when a GAN fails to capture the "periphery" or rare variations of a class. The authors test this by training a classifier entirely on GAN-generated data and evaluating its performance on a hold-out set of real data. If the GAN perfectly captures the true distribution, the classifier should generalize seamlessly. A drop in accuracy indicates shifted decision boundaries.
Conceptual illustration of boundary distortion. (a) Eigenvalues showing the skewed spectrum of GAN data versus true data. (b) Visualization of how boundary distortion in synthetic data leads to shifted decision boundaries.

Experimental Setup

The study evaluates five popular GAN architectures (DCGAN, WGAN, ALI, BEGAN, and Improved GAN) across two major benchmarks:

  • CelebA: A large-scale dataset of face attributes, focusing on subsets like "Smiling," "Makeup," "Bangs," and "Male."
  • LSUN: The Large-scale Scene Understanding dataset, specifically utilizing the "Bedroom," "Kitchen," and "Classroom" categories.

Key Findings

The classification-based evaluation reveals several critical insights into the true diversity of GAN outputs:

  • Visual Quality Does Not Equal Diversity: A GAN can produce stunning, high-resolution images while still suffering from massive mode collapse. For instance, while BEGAN produces highly realistic faces, it exhibited some of the most severe diversity loss in the study.
  • The "Sample Efficiency" Paradox: The most striking finding emerges when comparing GAN data to down-sampled real data. Classifiers trained on hundreds of thousands of GAN images often perform no better than classifiers trained on just 100–500 real images. This suggests the "effective" variety in a GAN distribution is orders of magnitude smaller than the original training set.
  • Universal Mode Collapse and Generalization Gap: Every GAN tested exhibited significant mode collapse and boundary distortion. In the LSUN Bedroom/Kitchen task, a ResNet trained on real data achieved roughly 99% accuracy. However, when trained on synthetic data from various GANs, accuracy on real test data plummeted to between 51% and 56%—barely better than random guessing.
  • Oversampling Is Ineffective: Generating ten times more synthetic data to train the classifier did not significantly improve performance. This confirms that the underlying issue is a fundamental lack of diversity in the GAN's learned distribution, not merely a lack of generated samples.
Quantitative results demonstrating the massive generalization gap. Classifiers trained on hundreds of thousands of GAN images perform similarly to those trained on only a few hundred true images.

Limitations and Future Directions

While this framework provides a robust quantitative measure of diversity, it relies on datasets with existing attribute labels (like CelebA) to train the initial annotator and define the classification tasks. Additionally, the exact accuracy drops can be sensitive to the specific classifier architecture used, although the relative trends remain consistent across models.

Ultimately, this classification-based approach offers a rigorous new standard metric for the generative modeling community, moving beyond simple visual inspection. The authors suggest this framework could easily be extended to evaluate other generative models, such as Variational Autoencoders (VAEs), to better understand their distributional fidelity.

Summary updated

Abstract

A basic, and still largely unanswered, question in the context of Generative Adversarial Networks (GANs) is whether they are truly able to capture all the fundamental characteristics of the distrib...
BibTeX
@inproceedings{santurkar2018classification,
  author = {Shibani Santurkar and Ludwig Schmidt and Aleksander Madry},
  title = {{A Classification-Based Study of Covariate Shift in GAN Distributions}},
  booktitle = {International Conference on Machine Learning},
  year = {2018},
  url = {https://openreview.net/forum?id=SyNiVo-OWr},
}
Figure