Lost in Translation: GANs' Inability to Generate Simple Probability Distributions

Debanjan Dutta, Anish Chakrabarty, Swagatam DasICLR 2024 · OpenReview

Generative Adversarial Networks (GANs) have marked a triumph in generative modeling, demonstrating an impeccable capacity to mimic observations from unknown, highly complex probability distributions. Because GANs routinely simulate data rich in semantic information—such as high-resolution images, text, and audio—out of random noise, it is natural to assume they can easily handle simpler tasks. However, recent research challenges this assumption, revealing that GANs are surprisingly inept at generating basic 1D probability distributions.

The "Simple" Problem GANs Can't Solve

In deep learning, it is often assumed that over-parameterized models will find simple problems trivial. One would expect GANs to master the "inverse transform sampling" method, the bedrock of statistical simulation where a model transforms simple noise into a target distribution.

The authors investigate whether there is a lower limit to the complexity a target distribution must possess for a GAN to estimate it accurately. Specifically, they examine whether GANs can induce simple statistical transformations, such as shifting the mean or scaling the variance of a standard Normal distribution.

Experimental Setup

To ensure that any observed failures were not due to a lack of model capacity or simple overfitting, the researchers rigorously tested three popular GAN variants: Vanilla GAN, Wasserstein GAN (WGAN), and Least Squares GAN (LSGAN).

The experimental design included:

  • 2-deep Generators: Theoretically proven to be optimal for approximating radial functions.
  • Multiple Activations: Testing ReLU, Leaky-ReLU, and Tanh to rule out non-linearity choices as the culprit.
  • Regularization: Applying Dropout to prevent overfitting.
  • Statistical Rigor: Moving beyond visual inspection by employing formal statistical tests, including the Anderson-Darling (AD) and Kolmogorov-Smirnov (KS) tests for normality, as well as specific tests for equality of mean and variance.

The models were tasked with mapping input noise (Uniform or Standard Normal) to simple target distributions, such as a standard Normal distribution $N(0, 1)$, a translated Normal distribution $N(-200, 1)$, and a translated/scaled Normal distribution $N(-200, 25)$.

Disillusioning Results

The results across all architectures were surprisingly poor. Despite the theoretical simplicity of the task, the GANs failed to perform basic statistical mappings.

The "Semantic Crutch" Hypothesis

The authors hypothesize that GANs function more as "pattern matchers" than true "distribution learners." GANs excel when a distribution is regular or contains intrinsic semantic patterns (like the spatial features of a human face). However, when the target is "information-poor"—such as a simple, smooth bell curve—the GAN lacks the semantic anchors needed to guide its adversarial training process.

Deceptive Convergence and Structural Failure

In many of the experiments, the loss functions for both the generator and discriminator appeared to converge, typically a signal of successful training. Yet, the resulting samples failed every formal statistical test. All three architectures failed the AD and KS tests for normality at a 5% significance level. Furthermore, when tasked with simply shifting a distribution to a mean of -200 (a basic affine transformation $y = ax + b$ that is trivial for a neural network to represent), every model failed the test for equality of means.

Qualitative Mess

Visually, the generated data further corroborated these failures. Instead of forming smooth Gaussian curves, the histograms of the generated data appeared spiky, clumpy, and fragmented.

Fragmented Histograms

Similarly, Quantile-Quantile (QQ) plots, which should form a straight diagonal line if the generated distribution perfectly matches the target Normal distribution, showed significant deviations, particularly at the tails.

Deviating QQ Plots

Conclusion and Future Directions

This research highlights a fundamental limitation in the adversarial training paradigm: a model's ability to generate complex, high-dimensional data does not guarantee its competence in basic statistical transformations.

The authors call for the development of a formal, quantitative measure of "semantic information." Such a metric could help practitioners predict whether a GAN is the appropriate tool for a specific simulation task. Future work may also explore whether significantly deeper architectures or alternative loss formulations can overcome this "translation gap," or if this inability to learn simple, information-poor distributions is an inherent flaw in GANs.

Summary updated

Abstract

Since its inception, Generative Adversarial Networks (GAN) have marked a triumph in generative modeling. Its impeccable capacity to mimic observations from unknown probability distributions has positioned it as a widely used simulation tool. In typical applications, GANs find themselves simulating data rich in semantic information such as images or text out of random noise. As such, it is reasonable to expect that large parametric models such as GANs must be able to estimate standard theoretical probability densities with ease. In this paper, based on a series of disillusioning experimental findings, we show that GANs often fail to induce the simplest of statistical transformations between distributions. For example, starting with a standard Gaussian noise, GANs with 2-deep generators are unable to perform a positional translation. Supporting theoretical tests on generated data further corroborates our rather unsettling conclusions.
BibTeX
@inproceedings{dutta2024lost,
  author = {Debanjan Dutta and Anish Chakrabarty and Swagatam Das},
  title = {{Lost in Translation: GANs' Inability to Generate Simple Probability Distributions}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=MUOmyZMEd4},
}
Figure