Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs

Alexia Jolicoeur-Martineau, Ioannis Mitliagkas

Abstract

A popular heuristic for improved performance in Generative adversarial networks (GANs) is to use some form of gradient penalty on the discriminator. This gradient penalty was originally motivated by a Wasserstein distance formulation. However, the use of gradient penalty in other GAN formulations is not well motivated. We present a unifying framework of expected margin maximization and show that a wide range of gradient-penalized GANs (e.g., Wasserstein, Standard, Least-Squares, and Hinge GANs) can be derived from this framework. Our results imply that employing gradient penalties induces a large-margin classifier (thus, a large-margin discriminator in GANs). We describe how expected margin maximization helps reduce vanishing gradients at fake (generated) samples, a known problem in GANs. From this framework, we derive a new $L^\infty$ gradient norm penalty with Hinge loss which generally produces equally good (or better) generated output in GANs than $L^2$-norm penalties (based on the Fr'echet Inception Distance).

Introduction

Generative Adversarial Networks (GANs) are powerful but notoriously difficult to train. A common technique to stabilize training is the use of "gradient penalties" on the discriminator. While originally introduced to enforce the Lipschitz constraint required by Wasserstein GANs (WGANs), these penalties have proven effective across various GAN losses, such as Cross-Entropy and Hinge loss, often without a rigorous theoretical justification. This paper bridges this gap by connecting gradient-penalized GANs to the principles of Support Vector Machines (SVMs), specifically through the lens of expected margin maximization.

The Framework of Expected Margin Maximization

The core contribution of this work is the unification of various GAN formulations under a maximum-margin classification framework. The authors define the geometric margin $\gamma(x)$ of a sample as its distance to the decision boundary. For complex neural networks, this margin can be approximated using a first-order Taylor expansion: $$\gamma(x) \approx \frac{|f(x)|}{||\nabla_x f(x)||_q}$$ By minimizing a loss function that includes a gradient penalty—$\min_f \mathbb{E}[L(yf(x)) + \lambda g(||\nabla_x f(x)||_q)]$—the discriminator effectively acts as a Maximum Margin Classifier. This process pushes the distributions of real and fake data apart while centering the decision boundary, which helps mitigate the vanishing gradient problem by ensuring the generator receives a consistent signal.

Visualization of decision boundaries (a) and gradient peaks (b). Margin maximization ensures the gradient does not vanish near samples, providing a learning signal for the generator.

Dual Norms and the $L^\infty$ Penalty

A significant insight from this framework is the relationship between the margin geometry and the gradient norm penalty. Specifically, maximizing an $L^p$ margin corresponds to penalizing the $L^q$ norm of the gradient, where $1/p + 1/q = 1$. While most existing methods use an $L^2$ penalty, the authors propose that an $L^\infty$ penalty (corresponding to an $L^1$ margin) might be more robust to outliers—a common occurrence in the early stages of GAN training.

Empirical Results

The authors evaluated their framework on toy datasets like the Swiss-roll and benchmarks like CIFAR-10. Experimental results on the Swiss-roll dataset confirmed that gradient penalties increase the expected margin by 2 to 3 times compared to unpenalized models.

On the CIFAR-10 image generation task, the proposed HingeGAN with an $L^\infty$ gradient penalty significantly outperformed standard approaches. As shown in the table below, it achieved a Fréchet Inception Distance (FID) of 27.1, which is superior to the 37.6 achieved by WGAN-GP and the 33.9 of HingeGAN with an $L^2$ penalty.

FID scores on CIFAR-10 for different GAN architectures and gradient penalties. The $L^\infty$ penalty with HingeGAN achieves the best performance (27.1).

Conclusion

This research provides a theoretical foundation for the success of gradient penalties in GANs, framing the discriminator as a large-margin classifier. By leveraging the connection between dual norms and margin geometry, the authors introduced a new $L^\infty$ penalty that enhances image generation quality. While the current derivation relies on local approximations, this framework opens new avenues for designing discriminators that directly maximize specific margins to improve GAN stability and performance.

Summary updated

Abstract

A popular heuristic for improved performance in Generative adversarial networks (GANs) is to use some form of gradient penalty on the discriminator. This gradient penalty was originally motivated by a Wasserstein distance formulation. However, the use of gradient penalty in other GAN formulations is not well motivated. We present a unifying framework of expected margin maximization and show that a wide range of gradient-penalized GANs (e.g., Wasserstein, Standard, Least-Squares, and Hinge GANs) can be derived from this framework. Our results imply that employing gradient penalties induces a large-margin classifier (thus, a large-margin discriminator in GANs). We describe how expected margin maximization helps reduce vanishing gradients at fake (generated) samples, a known problem in GANs. From this framework, we derive a new $L^\infty$ gradient norm penalty with Hinge loss which generally produces equally good (or better) generated output in GANs than $L^2$-norm penalties (based on the Fr\'echet Inception Distance).
BibTeX
@misc{jolicoeurmartineau2019connections,
  author = {Alexia Jolicoeur-Martineau and Ioannis Mitliagkas},
  title = {{Connections between Support Vector Machines, Wasserstein distance and gradient-penalty GANs}},
  year = {2019},
  eprint = {1910.06922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.06922},
}
Figure