On the Anomalous Generalization of GANs

Jinchen Xuan, Yunchang Yang, Ze Yang, Di He, Liwei WangarXiv 2019 · arXiv

Introduction

Generative Adversarial Networks (GANs) have set the standard for high-fidelity image synthesis. Despite their success, they often exhibit "anomalous generalization"—a failure to learn simple structural rules, such as the exact number of objects in a scene. For instance, a GAN trained on a dataset where every image contains exactly two rectangles might generate images with zero, one, or three rectangles. This paper investigates the root causes of this phenomenon, identifying sample insufficiency and discriminator vulnerabilities as primary culprits, and introduces two regularization techniques to address them.

The Mechanics of Anomalous Generalization

The authors argue that the gap between GAN theory and practical implementation leads to these failures. Two specific issues are highlighted:

1. Sample Insufficiency and Variance

In practice, GANs are trained using small mini-batches. Theoretical analysis using a WGAN model reveals that small batch sizes lead to high variance in the generator's parameter updates. This noise prevents the generator from "locking in" on the correct data distribution, making training dynamics unstable.

2. Discriminator Vulnerability to Averages

Discriminators, often built with ReLU activations, are prone to a mathematical trap: if they recognize two images as "real," they are highly likely to assign a high score to the pixel-wise average of those images. To a human, the average of two faces looks like a blurry "ghost" image; to a discriminator, it appears as a high-scoring success. This creates a "blind spot" that the generator exploits.

Figure 1: The "counting" problem in a geometry dataset. Vanilla GANs fail to capture the rule of exactly two rectangles, whereas the proposed SC+PCR methods achieve near-perfect accuracy.

Proposed Solutions

The researchers propose two straightforward regularization methods to improve generalization:

Sample Correction (SC)

To mitigate the instability of small batches, SC modifies the discriminator's training signal. If the generator produces a "perfect" sample that aligns with the target distribution, it is removed from the discriminator's negative batch. This prevents the discriminator from penalizing the generator for correct outputs, reducing contradictory gradients.

Pixel-wise Combination Regularization (PCR)

PCR explicitly teaches the discriminator to reject "ghost" images. The authors generate these anomalous samples—via pixel-wise averages or logical operations on training data—and feed them to the discriminator as negative samples. This forces the model to learn structural boundaries rather than just memorizing textures.

Figure 4: Illustrating the pixel-wise combination problem. Discriminators are often fooled by averages or logical combinations of training images, leading to anomalous generated outputs.

Experimental Evaluation and Results

The methods were tested on a synthetic Geometry Dataset and real-world benchmarks including CIFAR10, CelebA, and M-IMAGENET.

  • Solving the Counting Problem: On the geometry dataset, the vanilla GAN's accuracy in generating the correct number of rectangles was low. With SC+PCR, accuracy jumped to 99%.
  • Improved Image Quality: PCR significantly improved the Fréchet Inception Distance (FID) scores. For WGAN-GP on CIFAR10, FID improved by 30%. On M-IMAGENET, improvements reached up to 28.6% for the SAGAN architecture.
  • Visual Fidelity: Qualitative results on CelebA showed that SC helped prevent the generator from collapsing into noise, producing clearer facial details.

Limitations and Future Work

A primary challenge for Sample Correction (SC) is defining a "measure of reality" for complex, unlabeled natural images. While counting rectangles is simple, evaluating a face is not. The authors used L2 distance to the nearest training sample as a proxy, but suggest that more robust measures are needed. Additionally, the effectiveness of PCR can vary depending on the overlap within the dataset's manifold.

Summary updated

Abstract

Generative models, especially Generative Adversarial Networks (GANs), have received significant attention recently. However, it has been observed that in terms of some attributes, e.g. the number of simple geometric primitives in an image, GANs are not able to learn the target distribution in practice. Motivated by this observation, we discover two specific problems of GANs leading to anomalous generalization behaviour, which we refer to as the sample insufficiency and the pixel-wise combination. For the first problem of sample insufficiency, we show theoretically and empirically that the batchsize of the training samples in practice may be insufficient for the discriminator to learn an accurate discrimination function. It could result in unstable training dynamics for the generator, leading to anomalous generalization. For the second problem of pixel-wise combination, we find that besides recognizing the positive training samples as real, under certain circumstances, the discriminator could be fooled to recognize the pixel-wise combinations (e.g. pixel-wise average) of the positive training samples as real. However, those combinations could be visually different from the real samples in the target distribution. With the fooled discriminator as reference, the generator would obtain biased supervision further, leading to the anomalous generalization behaviour. Additionally, in this paper, we propose methods to mitigate the anomalous generalization of GANs. Extensive experiments on benchmark show our proposed methods improve the FID score up to 30\% on natural image dataset.
BibTeX
@misc{xuan2019anomalous,
  author = {Jinchen Xuan and Yunchang Yang and Ze Yang and Di He and Liwei Wang},
  title = {{On the Anomalous Generalization of GANs}},
  year = {2019},
  eprint = {1909.12638},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.12638},
}
Figure