A domain agnostic measure for monitoring and evaluating GANs

Paulina Grnarova, Kfir Y. Levy, Aurélien Lucchi, Nathanaël Perraudin, Ian Goodfellow, Thomas Hofmann, Andreas Krause

Abstract: Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be determined by simply inspecting the generator and discriminator loss curves as they behave non-intuitively. We leverage the notion of duality gap from game theory to propose a measure that addresses both (i) and (ii) at a low computational cost. Extensive experiments show the effectiveness of this measure to rank different GAN models and capture the typical GAN failure scenarios, including mode collapse and non-convergent behaviours. This evaluation metric also provides meaningful monitoring on the progression of the loss during training. It highly correlates with FID on natural image datasets, and with domain specific scores for text, sound and cosmology data where FID is not directly suitable. In particular, our proposed metric requires no labels or a pretrained classifier, making it domain agnostic.

The Challenge of GAN Monitoring

Training Generative Adversarial Networks (GANs) is notoriously difficult because their loss curves are non-intuitive and often fail to signal convergence. In standard deep learning, a decreasing validation loss provides a clear indicator of model improvement. In contrast, GANs are structured as a two-player minimax game where a generator and a discriminator compete. Consequently, their individual loss curves often oscillate wildly or remain flat, even as the generator produces increasingly realistic samples. This lack of a "meaningful curve" makes it nearly impossible for researchers to determine optimal stopping points or detect failures like mode collapse without constant manual inspection.

Figure 1: Comparison of evaluation metrics. Likelihood-based models offer intuitive loss curves, whereas standard GAN losses (Minmax) are uninformative. The Duality Gap provides a meaningful curve for convergence detection and model ranking.

Furthermore, existing evaluation metrics like the Fréchet Inception Distance (FID) are domain-specific. They rely on pre-trained classifiers—usually trained on ImageNet—to judge quality. This dependency renders them ineffective for non-image domains such as music, text, or scientific data. There is a critical need for a universal, domain-agnostic metric.

The Duality Gap Framework

This paper introduces the Duality Gap (DG) as a computationally efficient proxy for GAN performance. Rooted in game theory, the duality gap measures how far a pair of players is from a Nash Equilibrium—the state where neither player can improve their position by changing strategies. In the context of GANs, a duality gap of zero indicates that the generator has perfectly captured the underlying data distribution.

Theoretical analysis shows that the DG is lower-bounded by the Jensen-Shannon (JS) divergence between real and generated data, providing a mathematical guarantee that a low DG translates to high-quality samples. To make this practical, the authors propose an efficient estimation method: training a separate "adversary" for a small, fixed number of steps, initialized with the current model's weights. This approach avoids the impossible task of finding an absolute global optimum while still providing a robust signal.

Empirical Validation and Stability

The effectiveness of the Duality Gap was tested across diverse data geometries, including toy datasets like rings, spirals, and grids. As shown in the experimental results, the DG reliably distinguishes between stable training and failure modes. In stable regimes, the DG drops and plateaus as the model converges. In unstable regimes—where the model might experience mode collapse or divergence—the DG remains high or spikes significantly, providing the "test loss" signal that GANs have historically lacked.

Figure 2: Duality Gap monitoring on toy datasets. The metric successfully distinguishes between stable convergence (top row) and unstable training/mode collapse (bottom row) across various data geometries.

Key Results Across Domains

The metric's "agnostic" nature was validated across several complex domains:

  • Images: On CIFAR-10, the metric showed a Pearson correlation of up to 0.94 with FID, tracking image quality without needing a pre-trained ImageNet model.
  • Cosmology: In N-body simulations, the DG correlated strongly with physical statistics like mass histograms, where FID typically fails due to the non-natural nature of the data.
  • Audio and Text: The metric successfully ranked models in audio synthesis and synthetic text generation (SeqGAN), proving its versatility.

Conclusion

The Duality Gap offers a robust, domain-agnostic solution for both monitoring GAN training and ranking different architectures. By providing a meaningful curve that correlates with sample quality and physical reality, it allows for automated stopping criteria and hyperparameter tuning without the need for labels or external classifiers. While sensitive to the optimization of the internal "adversary," it represents a significant step toward making GAN training as predictable as standard supervised learning.

Summary updated

Abstract

Generative Adversarial Networks (GANs) have shown remarkable results in modeling complex distributions, but their evaluation remains an unsettled issue. Evaluations are essential for: (i) relative assessment of different models and (ii) monitoring the progress of a single model throughout training. The latter cannot be determined by simply inspecting the generator and discriminator loss curves as they behave non-intuitively. We leverage the notion of duality gap from game theory to propose a measure that addresses both (i) and (ii) at a low computational cost. Extensive experiments show the effectiveness of this measure to rank different GAN models and capture the typical GAN failure scenarios, including mode collapse and non-convergent behaviours. This evaluation metric also provides meaningful monitoring on the progression of the loss during training. It highly correlates with FID on natural image datasets, and with domain specific scores for text, sound and cosmology data where FID is not directly suitable. In particular, our proposed metric requires no labels or a pretrained classifier, making it domain agnostic.
BibTeX
@inproceedings{grnarova2019domain,
  author = {Paulina Grnarova and Kfir Y. Levy and Aurélien Lucchi and Nathanaël Perraudin and Ian Goodfellow and Thomas Hofmann and Andreas Krause},
  title = {{A domain agnostic measure for monitoring and evaluating GANs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=Zcddi_yWLs},
}
Figure