Introduction
Generative Adversarial Networks (GANs) have set the standard for high-fidelity image synthesis. Despite their success, they often exhibit "anomalous generalization"—a failure to learn simple structural rules, such as the exact number of objects in a scene. For instance, a GAN trained on a dataset where every image contains exactly two rectangles might generate images with zero, one, or three rectangles. This paper investigates the root causes of this phenomenon, identifying sample insufficiency and discriminator vulnerabilities as primary culprits, and introduces two regularization techniques to address them.
The Mechanics of Anomalous Generalization
The authors argue that the gap between GAN theory and practical implementation leads to these failures. Two specific issues are highlighted:
1. Sample Insufficiency and Variance
In practice, GANs are trained using small mini-batches. Theoretical analysis using a WGAN model reveals that small batch sizes lead to high variance in the generator's parameter updates. This noise prevents the generator from "locking in" on the correct data distribution, making training dynamics unstable.
2. Discriminator Vulnerability to Averages
Discriminators, often built with ReLU activations, are prone to a mathematical trap: if they recognize two images as "real," they are highly likely to assign a high score to the pixel-wise average of those images. To a human, the average of two faces looks like a blurry "ghost" image; to a discriminator, it appears as a high-scoring success. This creates a "blind spot" that the generator exploits.
Proposed Solutions
The researchers propose two straightforward regularization methods to improve generalization:
Sample Correction (SC)
To mitigate the instability of small batches, SC modifies the discriminator's training signal. If the generator produces a "perfect" sample that aligns with the target distribution, it is removed from the discriminator's negative batch. This prevents the discriminator from penalizing the generator for correct outputs, reducing contradictory gradients.
Pixel-wise Combination Regularization (PCR)
PCR explicitly teaches the discriminator to reject "ghost" images. The authors generate these anomalous samples—via pixel-wise averages or logical operations on training data—and feed them to the discriminator as negative samples. This forces the model to learn structural boundaries rather than just memorizing textures.
Experimental Evaluation and Results
The methods were tested on a synthetic Geometry Dataset and real-world benchmarks including CIFAR10, CelebA, and M-IMAGENET.
- Solving the Counting Problem: On the geometry dataset, the vanilla GAN's accuracy in generating the correct number of rectangles was low. With SC+PCR, accuracy jumped to 99%.
- Improved Image Quality: PCR significantly improved the Fréchet Inception Distance (FID) scores. For WGAN-GP on CIFAR10, FID improved by 30%. On M-IMAGENET, improvements reached up to 28.6% for the SAGAN architecture.
- Visual Fidelity: Qualitative results on CelebA showed that SC helped prevent the generator from collapsing into noise, producing clearer facial details.
Limitations and Future Work
A primary challenge for Sample Correction (SC) is defining a "measure of reality" for complex, unlabeled natural images. While counting rectangles is simple, evaluating a face is not. The authors used L2 distance to the nearest training sample as a proxy, but suggest that more robust measures are needed. Additionally, the effectiveness of PCR can vary depending on the overlap within the dataset's manifold.
Summary updated