Gradient descent GAN optimization is locally stable

Vaishnavh Nagarajan, J. Zico Kolter

Introduction

Training Generative Adversarial Networks (GANs) is notoriously difficult. Early theoretical work suggested that GANs should naturally converge, but this analysis relied on unrealistic assumptions: that updates occur in "pure function space" and that the discriminator is fully optimized before every single generator update. In practice, models are trained using simultaneous gradient descent, where both networks take small steps concurrently. This discrepancy often leads to training instabilities, such as mode collapse (where the generator produces limited varieties of outputs) or oscillations (where the networks chase each other endlessly without improving).

This paper bridges the gap between theory and practice by providing the first rigorous proof that simultaneous gradient descent in GANs is locally stable under specific conditions. Furthermore, it reveals why certain variants like Wasserstein GANs (WGANs) are prone to non-convergent "limit cycles," and introduces a simple gradient-based regularization term to guarantee stability across GAN architectures.

Theoretical Framework: The ODE Method

To analyze the stability of simultaneous updates, the authors employ the Ordinary Differential Equation (ODE) method from stochastic approximation theory. By treating the discrete steps of gradient descent as a continuous dynamical system, they evaluate the system's stability by examining the Jacobian matrix (the matrix of all first-order partial derivatives) at the equilibrium point.

If the Jacobian is "Hurwitz"—meaning all its eigenvalues have negative real parts—the system is considered locally exponentially stable. In this state, the system will naturally converge toward the equilibrium if it gets close enough.

Key Theoretical Insights

  • Concave-Concave Dynamics: Most stable game-theoretic systems are convex-concave (e.g., a saddle point). The authors demonstrate that GANs are often concave-concave, which is typically an optimization nightmare. Surprisingly, they prove that traditional GANs can still achieve local stability despite this property (formalized in Theorem 3.1).
  • Zero Eigenvalues: Neural networks inherently possess redundant parameters, which lead to zero eigenvalues in the Jacobian. The authors show that as long as these correspond to a "subspace of equilibria" (equivalent solutions), the system remains asymptotically stable.
  • The WGAN "Cycling" Problem: The popular Wasserstein GAN (WGAN) lacks a specific curvature property ($f''(0) < 0$). Consequently, WGANs do not necessarily settle into an equilibrium. Instead, they can enter limit cycles, where the parameters orbit the true distribution indefinitely without converging.

Proposed Solution: Gradient-Based Regularization

To combat oscillations and force convergence, the authors introduce a novel regularization term added to the generator's objective:

$$ \eta \|\nabla_{\theta_D} V(\theta_D, \theta_G)\|^2 $$

This term penalizes the generator based on the norm of the discriminator's gradient. Using a "double backprop" approach, it effectively grants the generator a degree of "foresight" into the discriminator's subsequent reactions. This dampens oscillations and mathematically forces the Jacobian to become stable. The proposed method acts as a computationally efficient alternative to "Unrolled GANs," achieving comparable results with significantly less overhead (requiring only one extra gradient computation).

Experimental Validation

The authors validated their theoretical findings in controlled environments:

  • 2D Uniform Distribution: A simple linear-quadratic system used to visualize the orbits in WGAN optimization. While a vanilla WGAN orbits the 2D equilibrium endlessly, adding the proposed regularization term causes it to converge rapidly.
  • Mixture of 8 Gaussians: A classic 2D synthetic dataset used to test mode collapse.
  • Stacked MNIST: A variant of the MNIST dataset where three random digits are stacked into color channels, creating 1,000 distinct modes.

In both the 8-Gaussian and Stacked MNIST experiments, the regularized GAN successfully captured all modes, whereas traditional GANs and unregularized WGANs frequently collapsed to just a few modes.

Limitations and Future Work

While this paper provides foundational theoretical guarantees, the authors note several limitations:

  • Local vs. Global Convergence: The analysis is strictly local. It explains the dynamics when the generator is already "near" the true data distribution, but it does not guarantee that a GAN will find this equilibrium from a random initialization.
  • Realizability Assumption: The mathematical proof assumes the generator is sufficiently powerful to represent the true data distribution perfectly. In real-world scenarios with highly complex data and constrained models, this assumption may not hold.
  • Deep Network Complexity: The highly non-convex landscape of deep neural networks might introduce "bad" local equilibria that this local analysis cannot fully account for.

Future work will need to explore global convergence properties and investigate how these stability dynamics interact with various deep neural network architectures.

Summary updated

Abstract

Despite the growing prominence of generative adversarial networks (GANs), optimization in GANs is still a poorly understood topic. In this paper, we analyze the ``gradient descent'' form of GAN optimization, i.e., the natural setting where we simultaneously take small gradient steps in both generator and discriminator parameters. We show that even though GAN optimization does \emph{not} correspond to a convex-concave game (even for simple parameterizations), under proper conditions, equilibrium points of this optimization procedure are still \emph{locally asymptotically stable} for the traditional GAN formulation. On the other hand, we show that the recently proposed Wasserstein GAN can have non-convergent limit cycles near equilibrium. Motivated by this stability analysis, we propose an additional regularization term for gradient descent GAN updates, which \emph{is} able to guarantee local stability for both the WGAN and the traditional GAN, and also shows practical promise in speeding up convergence and addressing mode collapse.
BibTeX
@inproceedings{nagarajan2017gradient,
  author = {Vaishnavh Nagarajan and J. Zico Kolter},
  title = {{Gradient descent GAN optimization is locally stable}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=B1V17w-_-H},
}
Figure