A Domain Agnostic Measure for Monitoring and Evaluating GANs

Paulina Grnarova, Kfir Y. Levy, Aurélien Lucchi, Nathanaël Perraudin, Ian Goodfellow, Thomas Hofmann, Andreas Krause

TL;DR

Researchers have introduced a domain-agnostic metric for Generative Adversarial Networks (GANs) based on the game-theoretic concept of the duality gap. Unlike popular metrics like FID or Inception Score, this measure requires no pre-trained classifiers or labels, works across images, text, and audio, and finally provides a "test loss" curve that actually tells you if your training is succeeding or collapsing.

What Problem Does This Paper Address?

Evaluating GANs is notoriously difficult. In standard deep learning, we watch a loss curve go down; in GANs, the generator and discriminator are locked in a "zero-sum game" where one player's gain is the other's loss. Consequently, their individual loss curves often oscillate wildly or stay flat, offering zero insight into whether the model is actually improving.

Comparison of standard loss curves, uninformative GAN minmax curves, and the proposed Duality Gap curve.

To solve this lack of visibility, researchers usually turn to external metrics like the Inception Score (IS) or Fréchet Inception Distance (FID). However, these have three major flaws:

  1. Domain Specificity: They are designed for natural images and rely on features from a model trained on ImageNet. They don't work natively for music, text, or scientific data.
  2. Label Dependency: They often require labeled data or pre-trained classifiers.
  3. Snapshot Only: They are computationally expensive to run frequently, making them poor tools for real-time monitoring during training.

Methods: The Duality Gap

The authors propose using the Duality Gap (DG) from game theory. Since a GAN is a minimax game, "solving" it means finding a Nash Equilibrium—a state where neither the generator nor the discriminator can improve their position by changing their strategy alone.

The Duality Gap measures how far the current state is from that equilibrium. Specifically:

  • The Concept: It calculates the difference between the "worst-case" discriminator's performance against the current generator and the "worst-case" generator's performance against the current discriminator.
  • The Estimation: Because finding the absolute "worst-case" adversary is mathematically hard, the authors estimate it by taking the current models and training them for a few extra steps against each other on a separate "adversary-finding" data set.
  • The Split: To keep the estimate unbiased, they split the data into three sets: one for training the GAN, one for finding the "worst-case" adversaries, and one for the final evaluation.

Datasets

The authors validated their metric across a diverse range of data types to prove its domain-agnostic nature:

  • Toy Datasets: 2D mixtures of Gaussians (Ring, Spiral, and Grid patterns) to visualize mode collapse.
  • Images: MNIST (handwritten digits) and CelebA (faces, using the Progressive GAN architecture).
  • Cosmology: N-body simulation data, where FID is useless because the "images" aren't natural scenes.
  • Audio: Time-frequency representations of sound.
  • Text: Sequential data used with the SeqGAN architecture.

Key Insights and Ideas

  • Theoretical Grounding: The authors prove that the Duality Gap is an upper bound on the Jensen-Shannon Divergence (JSD) between the real and generated data distributions. If the DG is zero, the GAN has perfectly captured the data distribution.
  • Detecting Failure Modes: The metric is highly sensitive to both "stable" mode collapse (where the model settles on one type of output and stays there) and "unstable" collapse (where the model oscillates without improving).
Duality Gap effectively distinguishes between stable and unstable GAN training across toy datasets like Ring, Spiral, and Grid.
  • The "Test Loss" for GANs: For the first time, practitioners have a curve that behaves like a traditional loss curve. If the DG starts spiking or stops decreasing, you know exactly when to stop training or adjust hyperparameters.
  • Domain Agnosticism: Because the metric only looks at the GAN's internal objective function, it doesn't care if you are generating cat pictures, text sequences, or gravitational wave signals.

Key Results

  • High Correlation: On natural images, the Duality Gap and its sub-component (minimax loss) showed high Pearson correlation with FID (up to 0.94 for regularized models).
  • Cosmology Success: In the cosmology domain, the DG correlated strongly with domain-specific physical statistics (mass histograms and power spectral density), significantly outperforming FID.
  • Computational Efficiency: Estimating the DG is fast. In experiments on CIFAR-10, calculating the DG took about 27 seconds, compared to 120 seconds for FID.
  • Model Ranking: The metric successfully ranked different GAN variants (NS-GAN, WGAN-GP, etc.) in the same order as FID and Inception Score, proving its reliability as a ranking tool.

Limitations and Future Work

The primary limitation is that the Duality Gap is an estimate. Its accuracy depends on how well you can find the "worst-case" adversary during the estimation steps. If the optimization used to find the adversary fails, the DG might be underestimated.

The authors also note that while the metric is domain-agnostic, the architecture of the adversaries used to calculate the gap still needs to be appropriate for the data. Future work could involve automating the selection of these adversarial "probes" to make the metric even more plug-and-play for researchers in niche fields.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be determined by simply inspecting the generator and discriminator loss curves as they behave non-intuitively. We leverage the notion of duality gap from game theory to propose a measure that addresses both (i) and (ii) at a low computational cost. Extensive experiments show the effectiveness of this measure to rank different GAN models and capture the typical GAN failure scenarios, including mode collapse and non-convergent behaviours. This evaluation metric also provides meaningful monitoring on the progression of the loss during training. It highly correlates with FID on natural image datasets, and with domain specific scores for text, sound and cosmology data where FID is not directly suitable. In particular, our proposed metric requires no labels or a pretrained classifier, making it domain agnostic.
BibTeX
@inproceedings{grnarova2019domain,
  author = {Paulina Grnarova and Kfir Y. Levy and Aurélien Lucchi and Nathanaël Perraudin and Ian Goodfellow and Thomas Hofmann and Andreas Krause},
  title = {{A Domain Agnostic Measure for Monitoring and Evaluating GANs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=Zcddi_yWLs},
}
Figure