A Solvable High-Dimensional Model of GAN

Chuang Wang, Hong Hu, Yue M. Lu

Introduction

Training Generative Adversarial Networks (GANs) is notoriously difficult. Practitioners frequently encounter issues like "mode collapse"—where the generator produces a limited variety of outputs—and "oscillations," where the generator and discriminator chase each other endlessly without converging.

While researchers have attempted to model these dynamics theoretically, most previous frameworks assumed a "small learning rate" limit. In that simplified scenario, the mathematics become deterministic, effectively ignoring the inherent randomness (stochasticity) of the training process. This paper addresses that critical gap by studying GANs in a high-dimensional limit ($n \to \infty$), where the number of data features is massive. By doing so, the authors account for the noise in the system and reveal how it fundamentally dictates whether a GAN succeeds or fails. Ultimately, they demonstrate that background noise is not merely a nuisance, but a vital stabilizer that prevents training from spiraling into endless oscillations.

Methodology

The authors focus on a simplified, yet mathematically rigorous, setup to isolate the core dynamics of adversarial training:

  • The Data Model: They employ a "spiked covariance model," where the target data consists of a few important latent features hidden within high-dimensional Gaussian noise.
  • The GAN Architecture: The model uses a single-layer linear generator and a single-feature discriminator. While simpler than modern deep neural networks, this "toy model" accurately captures the fundamental competitive dynamics of GANs.
  • Two-Level Analysis:
    • Macroscopic: The authors track "summary statistics," such as the cosine similarity between the generator’s weights and the true data features. They prove that these macroscopic quantities converge to a deterministic process characterized by an Ordinary Differential Equation (ODE).
  • Microscopic: Simultaneously, they track the individual weights themselves. These remain stochastic and are described by a Stochastic Differential Equation (SDE).

Because this is a theoretical analysis, the study does not rely on standard image benchmarks like CIFAR-10 or ImageNet. Instead, it uses synthetic high-dimensional data generated from the spiked covariance model. This allows the authors to precisely control the signal-to-noise ratio and verify their mathematical proofs against numerical simulations without running the full training loop.

Key Insights

The Stabilizing Role of Noise The most striking finding of the paper is that noise acts as a double-edged sword. If the background noise is too strong, the GAN fails to recover any meaningful features. However, if the noise is too weak, the system becomes unstable and begins to oscillate. There is a precise, intermediate level of noise that is required to stabilize the training process.

Sequential Learning Even with a simple discriminator, the generator learns features sequentially. It first aligns with the strongest feature in the dataset, "deceives" the discriminator on that front, and then moves on to the next most prominent feature.

SDE vs. ODE Unlike previous work that treated weights as entirely deterministic, this paper shows that while the performance of the GAN (the macroscopic state) is predictable, the weights (the microscopic state) are fundamentally noisy. Ignoring this microscopic noise leads to an incomplete understanding of GAN stability.

Key Results

Based on learning rates and noise levels, the authors identified distinct phases of GAN training: Success, Failure, Oscillation, and Mode-Collapsing.

Macroscopic training dynamics showing success, oscillation, and failure phases.

As shown in the figure above, numerical simulations with 5,000 dimensions perfectly matched the trajectories predicted by their theoretical ODEs. The solid lines represent the deterministic ODE predictions, while the dots represent the empirical results from the simulations, confirming the accuracy of the high-dimensional limit analysis.

Furthermore, the authors derived a specific mathematical inequality that defines the "safe zone" for successful training. If the chosen learning rates and noise levels fall outside this bound, the GAN is mathematically guaranteed to either oscillate or collapse.

Limitations and Future Work

While this paper provides a rigorous foundation, it has limitations inherent to theoretical models:

  • Linearity: The model uses a linear generator. While necessary for an "exactly solvable" proof, real-world GANs rely on deep, non-linear networks.
  • Single-Feature Discriminator: The discriminator in this model only isolates one feature at a time. Analyzing multi-feature discriminators would require more complex techniques.

The authors view this work as a foundational building block. The natural next step is to extend these mean-field analysis techniques to multi-layer networks, investigating whether the "noise-as-stabilizer" insight holds true for complex, state-of-the-art architectures like BigGAN or StyleGAN.

Summary updated

Abstract

We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the high-dimensional limit. In particular, we prove that the macroscopic quantities measuring the quality of the training process converge to a deterministic process characterized by an ordinary differential equation (ODE), whereas the microscopic states containing all the detailed weights remain stochastic, whose dynamics can be described by a stochastic differential equation (SDE). This analysis provides a new perspective different from recent analyses in the limit of small learning rate, where the microscopic state is always considered deterministic, and the contribution of noise is ignored. From our analysis, we show that the level of the background noise is essential to the convergence of the training process: setting the noise level too strong leads to failure of feature recovery, whereas setting the noise too weak causes oscillation. Although this work focuses on a simple copy model of GAN, we believe the analysis methods and insights developed here would prove useful in the theoretical understanding of other variants of GANs with more advanced training algorithms.
BibTeX
@inproceedings{wang2019solvable,
  author = {Chuang Wang and Hong Hu and Yue M. Lu},
  title = {{A Solvable High-Dimensional Model of GAN}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=sSWqs5e3Pf6},
}
Figure