Gradient penalty from a maximum margin perspective

Alexia Jolicoeur‐Martineau, Ioannis Mitliagkas

Generative Adversarial Networks (GANs) are powerful tools for generating realistic data, but their training is notoriously unstable. To mitigate this, researchers often employ a "gradient penalty" (GP) on the discriminator. While originally motivated by the Wasserstein distance formulation, the effectiveness of GP across various GAN architectures (Standard, Least-Squares, Hinge) lacked a unifying explanation. This paper by Jolicoeur-Martineau and Mitliagkas proposes that gradient penalties function by inducing a large-margin classifier in the discriminator, providing a theoretical bridge between GANs and Support Vector Machines (SVMs).

The Problem: Vanishing Gradients and Heuristic Penalties

In the GAN framework, the discriminator $D$ and generator $G$ engage in a minimax game. A common failure mode occurs when the discriminator becomes too proficient, leading to vanishing gradients for the generator. Gradient penalties were introduced to constrain the discriminator's function space, ensuring smoother gradients. However, the theoretical justification for these penalties was often tied specifically to the 1-Lipschitz constraint of Wasserstein GANs. The authors seek to explain why GP improves performance even in non-Wasserstein settings.

A Unifying Framework: Expected Margin Maximization

The core contribution of this work is the reframing of gradient penalties as a form of expected margin maximization. By applying Taylor expansions, the authors demonstrate that the geometric margin—the distance from a sample to the decision boundary—is inversely proportional to the norm of the discriminator's gradient.

Visualization of decision boundaries and gradient behavior. (a) shows the margin between real and fake samples. (b) illustrates how different functions affect the gradient distribution.

Key components of their methodology include:

  • Generalized Objective: They propose a loss function that combines a margin-based loss with a gradient penalty, proving its equivalence to maximizing the expected margin.
  • The $L^\infty$ Gradient Penalty: Moving beyond the standard $L^2$ norm, the authors introduce an $L^\infty$ norm penalty. Based on dual norm theory, an $L^\infty$ penalty on the gradient corresponds to maximizing an $L^1$ margin, which tends to be more robust to outliers.
  • Hinge-style Constraints: Instead of an equality constraint (forcing the gradient norm to be exactly 1), they advocate for a one-sided penalty (a Hinge function) that only penalizes gradients exceeding a threshold.

Empirical Validation and Results

The authors tested their framework on synthetic datasets (Swiss-roll) and standard benchmarks like CIFAR-10 and the CAT dataset. The results demonstrate that the $L^\infty$ Hinge-GP significantly improves image quality as measured by the Fréchet Inception Distance (FID).

FID scores for various GAN architectures and gradient penalty types. The $L^\infty$ Hinge-GP (bottom right) achieves the best performance.

On CIFAR-10, the proposed $L^\infty$ Hinge-GP achieved an FID of 27.1, a substantial improvement over the 37.6 achieved by the standard WGAN-GP. Beyond FID scores, the authors observed that:

  • Margin Size: Applying a gradient penalty increased the expected margin by 2-3 times in toy experiments.
  • Stability: The large-margin property ensures that the discriminator provides informative gradients to the generator even for samples far from the boundary, effectively solving the vanishing gradient problem.

Conclusion

By viewing gradient penalties through the lens of maximum margin classification, this research provides a solid theoretical foundation for a widely used heuristic. The derivation of the $L^\infty$ penalty offers a practical improvement for GAN training, resulting in more stable models and higher-quality generated outputs.

Reference: Jolicoeur-Martineau, A., & Mitliagkas, I. (2020). Gradient penalty from a maximum margin perspective. arXiv:1910.06922v2.

Summary updated

Abstract

A popular heuristic for improved performance in Generative adversarial networks (GANs) is to use some form of gradient penalty on the discriminator. This gradient penalty was originally motivated by a Wasserstein distance formulation. However, the use of gradient penalty in other GAN formulations is not well motivated. We present a unifying framework of expected margin maximization and show that a wide range of gradient-penalized GANs (e.g., Wasserstein, Standard, Least-Squares, and Hinge GANs) can be derived from this framework. Our results imply that employing gradient penalties induces a large-margin classifier (thus, a large-margin discriminator in GANs). We describe how expected margin maximization helps reduce vanishing gradients at fake (generated) samples, a known problem in GANs. From this framework, we derive a new $L^\infty$ gradient norm penalty with Hinge loss which generally produces equally good (or better) generated output in GANs than $L^2$-norm penalties (based on the Fréchet Inception Distance).
BibTeX
@misc{jolicoeurmartineau2019gradient,
  author = {Alexia Jolicoeur‐Martineau and Ioannis Mitliagkas},
  title = {{Gradient penalty from a maximum margin perspective}},
  year = {2019},
  eprint = {1910.06922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.06922},
}
Figure