Empirical Analysis of Overfitting and Mode Drop in GAN Training

Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Vijay Chandrasekhar

Abstract

Generative Adversarial Networks (GANs) are renowned for their ability to synthesize high-fidelity images, yet they frequently suffer from two critical issues: overfitting (memorization of training data) and mode drop (failure to capture the full data distribution). This study challenges the conventional view that these are inherent flaws of the GAN objective. By systematically reducing the stochasticity in training—specifically through batch size manipulation and latent noise control—the authors demonstrate that GANs can achieve perfect recall of training data, effectively "memorizing" the set. This reveals a fundamental trade-off: stochasticity serves as a necessary regularizer against overfitting but is a primary driver of mode drop.

Introduction

The training of Generative Adversarial Networks (GANs) is traditionally viewed as a game-theoretic optimization where a generator learns to map a latent space to a data distribution by competing against a discriminator. While successful, GANs often exhibit "mode drop," where the generator ignores significant portions of the target distribution. Conversely, it has long been argued that GANs are resistant to traditional overfitting because the generator never directly observes the training samples.

Researchers from ASTAR and NTU Singapore investigate these phenomena through the lens of stochasticity. Their work, "Empirical Analysis of Overfitting and Mode Drop in GAN Training,"* posits that the inherent randomness in standard GAN training—arising from mini-batch sampling and random latent vector generation—is the hidden variable governing both generalization and mode coverage.

Methodology

To isolate the effects of randomness, the authors established a "deterministic" baseline using a standard DCGAN architecture with spectral normalization. In this configuration:

  1. Fixed Latent Vectors: A set of latent noise vectors was fixed to match the size of the training subset.
  2. Full-Batch Training: Extremely large batch sizes were employed, allowing the model to see the entire training subset in a single step, thereby eliminating sampling noise.

Stochasticity was then reintroduced systematically by varying the mini-batch size ($m$) from near-deterministic (3,200) to highly stochastic (50). Additionally, the impact of "noisy" latent codes was tested by adding random jitter to fixed input vectors.

Evaluation Metrics

The primary metric used was the 1-Nearest Neighbor (1-NN) test. By calculating the distance between generated images and their nearest neighbors in the training set (in both pixel and feature space), the authors could quantify:

  • Overfitting: Low distance indicates the generator is reproducing specific training samples.
  • Mode Drop: High recall (every training sample having a close generated match) indicates low mode drop.

Experimental Results

The study utilized three benchmark datasets: SVHN, CIFAR-10, and FFHQ, all downsampled to $32 \times 32$ pixels to accommodate large-batch training.

The Stochasticity Trade-off

The experiments revealed a clear inverse relationship between stochasticity and memorization. As training became more deterministic (larger batch sizes), the 1-NN distances dropped sharply. For the CIFAR-10 dataset, the pixel-wise error decreased from 28.31 in the stochastic setting to 4.90 in the deterministic setting.

Visual comparison of generated results across SVHN, CIFAR-10, and FFHQ datasets under varying levels of stochasticity.

In the most deterministic cases, the generator achieved "complete recall," successfully recreating every image in the training subset. This suggests that mode drop is not solely a consequence of the GAN loss function, but is heavily influenced by the noise introduced during training.

Visual Evidence of Memorization

Qualitative analysis confirms the quantitative findings. At high batch sizes, the generated images become visually indistinguishable from the training data. This effect is particularly striking in the FFHQ dataset, where the model transitions from generating generic faces to perfectly reconstructing specific individuals from the training set as stochasticity is reduced.

Detailed view of FFHQ face reconstructions, demonstrating the generator's capacity for high-fidelity memorization as training becomes more deterministic.

Discussion: Why Randomness Matters

The authors propose the "Superficial Explanations" theory to explain why stochasticity prevents overfitting. In small-batch training, the gradient signal is noisy. As the model nears convergence, this noise dominates, preventing the discriminator from providing the precise feedback needed for the generator to learn fine-grained details. Consequently, the generator settles for a "generalized" representation. In contrast, deterministic training provides a stable signal that allows the generator to find the exact mapping to training samples, leading to perfect memorization.

Conclusion

This empirical analysis demonstrates that GANs are "closet memorizers." The perceived resistance to overfitting in standard GANs is a byproduct of stochastic training rather than an inherent property of the adversarial objective. While randomness is essential for generalization, it is also a major contributor to mode drop. Future research should focus on finding an optimal balance—perhaps through adaptive stochasticity—to harness the full expressive power of GANs without sacrificing diversity or generalization.

Summary updated

Abstract

We examine two key questions in GAN training, namely overfitting and mode drop, from an empirical perspective. We show that when stochasticity is removed from the training procedure, GANs can overfit and exhibit almost no mode drop. Our results shed light on important characteristics of the GAN training procedure. They also provide evidence against prevailing intuitions that GANs do not memorize the training set, and that mode dropping is mainly due to properties of the GAN objective rather than how it is optimized during training.
BibTeX
@inproceedings{yazici2020empirical,
  author = {Yasin Yazici and Chuan-Sheng Foo and Stefan Winkler and Kim-Hui Yap and Vijay Chandrasekhar},
  title = {{Empirical Analysis of Overfitting and Mode Drop in GAN Training}},
  booktitle = {ICIP},
  year = {2020},
  doi = {10.1109/icip40778.2020.9191083},
}
Figure