Abstract
Generative Adversarial Networks (GANs) are renowned for their ability to synthesize high-fidelity images, yet they frequently suffer from two critical issues: overfitting (memorization of training data) and mode drop (failure to capture the full data distribution). This study challenges the conventional view that these are inherent flaws of the GAN objective. By systematically reducing the stochasticity in training—specifically through batch size manipulation and latent noise control—the authors demonstrate that GANs can achieve perfect recall of training data, effectively "memorizing" the set. This reveals a fundamental trade-off: stochasticity serves as a necessary regularizer against overfitting but is a primary driver of mode drop.
Introduction
The training of Generative Adversarial Networks (GANs) is traditionally viewed as a game-theoretic optimization where a generator learns to map a latent space to a data distribution by competing against a discriminator. While successful, GANs often exhibit "mode drop," where the generator ignores significant portions of the target distribution. Conversely, it has long been argued that GANs are resistant to traditional overfitting because the generator never directly observes the training samples.
Researchers from ASTAR and NTU Singapore investigate these phenomena through the lens of stochasticity. Their work, "Empirical Analysis of Overfitting and Mode Drop in GAN Training,"* posits that the inherent randomness in standard GAN training—arising from mini-batch sampling and random latent vector generation—is the hidden variable governing both generalization and mode coverage.
Methodology
To isolate the effects of randomness, the authors established a "deterministic" baseline using a standard DCGAN architecture with spectral normalization. In this configuration:
- Fixed Latent Vectors: A set of latent noise vectors was fixed to match the size of the training subset.
- Full-Batch Training: Extremely large batch sizes were employed, allowing the model to see the entire training subset in a single step, thereby eliminating sampling noise.
Stochasticity was then reintroduced systematically by varying the mini-batch size ($m$) from near-deterministic (3,200) to highly stochastic (50). Additionally, the impact of "noisy" latent codes was tested by adding random jitter to fixed input vectors.
Evaluation Metrics
The primary metric used was the 1-Nearest Neighbor (1-NN) test. By calculating the distance between generated images and their nearest neighbors in the training set (in both pixel and feature space), the authors could quantify:
- Overfitting: Low distance indicates the generator is reproducing specific training samples.
- Mode Drop: High recall (every training sample having a close generated match) indicates low mode drop.
Experimental Results
The study utilized three benchmark datasets: SVHN, CIFAR-10, and FFHQ, all downsampled to $32 \times 32$ pixels to accommodate large-batch training.
The Stochasticity Trade-off
The experiments revealed a clear inverse relationship between stochasticity and memorization. As training became more deterministic (larger batch sizes), the 1-NN distances dropped sharply. For the CIFAR-10 dataset, the pixel-wise error decreased from 28.31 in the stochastic setting to 4.90 in the deterministic setting.
In the most deterministic cases, the generator achieved "complete recall," successfully recreating every image in the training subset. This suggests that mode drop is not solely a consequence of the GAN loss function, but is heavily influenced by the noise introduced during training.
Visual Evidence of Memorization
Qualitative analysis confirms the quantitative findings. At high batch sizes, the generated images become visually indistinguishable from the training data. This effect is particularly striking in the FFHQ dataset, where the model transitions from generating generic faces to perfectly reconstructing specific individuals from the training set as stochasticity is reduced.
Discussion: Why Randomness Matters
The authors propose the "Superficial Explanations" theory to explain why stochasticity prevents overfitting. In small-batch training, the gradient signal is noisy. As the model nears convergence, this noise dominates, preventing the discriminator from providing the precise feedback needed for the generator to learn fine-grained details. Consequently, the generator settles for a "generalized" representation. In contrast, deterministic training provides a stable signal that allows the generator to find the exact mapping to training samples, leading to perfect memorization.
Conclusion
This empirical analysis demonstrates that GANs are "closet memorizers." The perceived resistance to overfitting in standard GANs is a byproduct of stochastic training rather than an inherent property of the adversarial objective. While randomness is essential for generalization, it is also a major contributor to mode drop. Future research should focus on finding an optimal balance—perhaps through adaptive stochasticity—to harness the full expressive power of GANs without sacrificing diversity or generalization.
Summary updated