A Solvable High-Dimensional Model of GAN

Chuang Wang, Hong Hu, Yue M. Lu

Abstract: We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the high-dimensional limit. In particular, we prove that the macroscopic quantities measuring the quality of the training process converge to a deterministic process characterized by an ordinary differential equation (ODE), whereas the microscopic states containing all the detailed weights remain stochastic, whose dynamics can be described by a stochastic differential equation (SDE). This analysis provides a new perspective different from recent analyses in the limit of small learning rate, where the microscopic state is always considered deterministic, and the contribution of noise is ignored. From our analysis, we show that the level of the background noise is essential to the convergence of the training process: setting the noise level too strong leads to failure of feature recovery, whereas setting the noise too weak causes oscillation. Although this work focuses on a simple copy model of GAN, we believe the analysis methods and insights developed here would prove useful in the theoretical understanding of other variants of GANs with more advanced training algorithms.

Introduction

Despite the massive success of Generative Adversarial Networks (GANs) in generating realistic data, our theoretical understanding of their convergence remains limited. Most existing analyses assume a "small learning rate" limit, treating the training process as entirely deterministic. However, real-world training is inherently noisy. This research addresses the gap by analyzing GAN dynamics in the high-dimensional limit ($n \to \infty$), seeking to understand how macroscopic performance metrics emerge from the stochastic movements of individual weights.

Theoretical Framework and Methods

The researchers employ a framework based on the scaling limits of high-dimensional exchangeable stochastic processes. They utilize a single-layer GAN architecture where "real" data follows a spiked covariance model—meaning a few meaningful features are hidden within high-dimensional Gaussian noise. This allows for precise control over the Signal-to-Noise Ratio (SNR) and dimensionality.

The core of the methodology is a two-level analysis:

  • Macroscopic Scale: Aggregate quantities, such as the cosine similarity between learned weights and ground-truth features, are shown to converge to a deterministic Ordinary Differential Equation (ODE).
  • Microscopic Scale: Individual weights are shown to remain stochastic, following a Stochastic Differential Equation (SDE). This highlights that while overall performance might look predictable, the underlying weights continue to fluctuate.
Evolution of microscopic weight distributions (red dots) and their macroscopic characterization via ellipses at different time steps $t=0, 10, 100, 150$.

Key Insights: The Role of Noise

A counter-intuitive finding of this work is that noise can act as a stabilizer. If background noise is too weak, the competition between the generator and discriminator becomes volatile, leading to permanent oscillations. Conversely, if noise is too strong, the generator fails to recover features entirely.

The model also reveals "sequential learning": even with a simple discriminator, the generator can learn multiple features one after another. The discriminator identifies a feature, the generator learns to mimic it, and then the discriminator shifts its focus to the next available feature.

Results and Phase Transitions

The derived ODEs match experimental trajectories with high accuracy. The analysis identifies four distinct phases of GAN training:

  1. Success: Features are recovered, and the generator fools the discriminator.
  2. Failure: No features are learned due to low SNR.
  3. Oscillation: The system enters a limit cycle and never settles.
  4. Mode Collapse: Only a subset of features is recovered.
Comparison of training trajectories for macroscopic variables, illustrating successful convergence (left and right) versus oscillatory behavior (middle).

Furthermore, the authors provided a closed-form stability condition that guarantees the "perfect estimation" state is locally stable, providing a mathematical bound for successful training based on learning rates and SNR.

Conclusion and Future Directions

This work provides a new perspective on GAN stability by accounting for microscopic noise. While currently limited to a single-layer linear model, the analysis methods developed here offer a path toward understanding more complex, non-linear, and multi-layer GAN architectures. Future work aims to explore cases where the number of features scales with the data dimension, representing even more complex data distributions.

Summary updated

Abstract

We present a theoretical analysis of the training process for a single-layer GAN fed by high-dimensional input data. The training dynamics of the proposed model at both microscopic and macroscopic scales can be exactly analyzed in the high-dimensional limit. In particular, we prove that the macroscopic quantities measuring the quality of the training process converge to a deterministic process characterized by an ordinary differential equation (ODE), whereas the microscopic states containing all the detailed weights remain stochastic, whose dynamics can be described by a stochastic differential equation (SDE). This analysis provides a new perspective different from recent analyses in the limit of small learning rate, where the microscopic state is always considered deterministic, and the contribution of noise is ignored. From our analysis, we show that the level of the background noise is essential to the convergence of the training process: setting the noise level too strong leads to failure of feature recovery, whereas setting the noise too weak causes oscillation. Although this work focuses on a simple copy model of GAN, we believe the analysis methods and insights developed here would prove useful in the theoretical understanding of other variants of GANs with more advanced training algorithms.
BibTeX
@inproceedings{wang2019solvable,
  author = {Chuang Wang and Hong Hu and Yue M. Lu},
  title = {{A Solvable High-Dimensional Model of GAN}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=sSWqs5e3Pf6},
}
Figure