Three-Player Wasserstein GAN via Amortised Duality

Nhan Dam, Quan Hoang, Trung Le, Tu Dinh Nguyen, Hung Bui, Dinh PhungIJCAI 2019 · OpenReview

Abstract

We propose a new formulation for learning generative adversarial networks (GANs) using optimal transport cost (the general form of Wasserstein distance) as the objective criterion to measure the dissimilarity between target distribution and learned distribution. Our formulation is based on the general form of the Kantorovich duality which is applicable to optimal transport with a wide range of cost functions that are not necessarily metric. To make optimising this duality form amenable to gradient-based methods, we employ a function that acts as an amortised optimiser for the innermost optimisation problem. Interestingly, the amortised optimiser can be viewed as a mover since it strategically shifts around data points. The resulting formulation is a sequential min-max-min game with 3 players: the generator, the critic, and the mover where the new player, the mover, attempts to fool the critic by shifting the data around. Despite involving three players, we demonstrate that our proposed formulation can be trained reasonably effectively via a simple alternative gradient learning strategy. Compared with the existing Lipschitz-constrained formulations of Wasserstein GAN on CIFAR-10, our model yields significantly better diversity scores than weight clipping and comparable performance to gradient penalty method.

Introduction

Generative Adversarial Networks (GANs) are powerful tools for modeling complex data distributions but are notoriously difficult to train. Common issues include "mode collapse," where the generator produces a limited variety of samples, and vanishing gradients. The Wasserstein GAN (WGAN) was introduced to mitigate these problems by using the Earth Mover’s distance as a smoother objective. However, WGANs require the "critic" function to be 1-Lipschitz continuous. Enforcing this constraint typically involves suboptimal "hacks" like weight clipping, which restricts model capacity, or gradient penalties, which are computationally intensive.

This paper introduces 3P-WGAN, a framework that bypasses these restrictive constraints by introducing a third neural network called the "mover." By leveraging amortized optimization to solve the Kantorovich duality, 3P-WGAN creates a stable three-player game that achieves high sample diversity without the need for traditional Lipschitz enforcement.

Methodology: The Three-Player Game

The core innovation of 3P-WGAN is the reformulation of the Kantorovich duality into a sequential min-max-min game involving three distinct players:

  1. The Generator ($g$): Learns to map latent noise to realistic data samples.
  2. The Critic ($f$): A standard neural network that learns to distinguish between real and generated data. Unlike standard WGANs, this critic is unconstrained.
  3. The Mover ($h$): Acts as an amortized optimizer. It learns to strategically shift real data points to "fool" the critic.

The mover is penalized by a transport cost (e.g., Euclidean distance), ensuring it only moves data minimally to regularize the critic. This prevents the critic from becoming too sharp, forcing it to learn more meaningful, global features of the data distribution. At equilibrium, the generator perfectly matches the data, and the mover becomes an identity function ($h(x) = x$).

Evolution of the 3P-WGAN training process on a synthetic 2D mixture of Gaussians, illustrating the mover's dynamics in aligning distributions.

Experimental Results

The authors evaluated 3P-WGAN on synthetic 2D data, CIFAR-10, and CelebA. The model demonstrated remarkable stability; in synthetic tests, it maintained convergence for thousands of epochs after reaching the optimal state.

Quantitative Performance

On the CIFAR-10 benchmark, 3P-WGAN achieved a Frechet Inception Distance (FID) of 28.8. This performance slightly outperforms WGAN with Gradient Penalty (WGAN-GP, FID 29.3) and significantly surpasses the original weight-clipped WGAN (FID 74.6). Crucially, the model yielded superior diversity scores, effectively avoiding the mode collapse issues that plague many adversarial frameworks.

Qualitative Quality

Visual inspection of generated samples confirms the model's ability to capture complex features. On CelebA, the model generated high-quality faces with significant variation in attributes. On CIFAR-10, the samples remain distinct and recognizable across all ten classes.

Representative samples generated by 3P-WGAN trained on the CIFAR-10 dataset, demonstrating high diversity and visual clarity.

Conclusion and Future Work

3P-WGAN offers a robust alternative to Lipschitz-constrained GANs. By shifting the mathematical burden of the Wasserstein distance to a mover network, the framework allows for more flexible critic architectures and more stable training dynamics. While the addition of a third network introduces some computational overhead and requires careful hyperparameter tuning (specifically for the cost scale parameter $\lambda$), the gains in sample diversity and training stability are significant. Future research may explore specialized cost functions for non-image data types like text or audio.

Summary updated

Abstract

We propose a new formulation for learning generative adversarial networks (GANs) using optimal transport cost (the general form of Wasserstein distance) as the objective criterion to measure the dissimilarity between target distribution and learned distribution. Our formulation is based on the general form of the Kantorovich duality which is applicable to optimal transport with a wide range of cost functions that are not necessarily metric. To make optimising this duality form amenable to gradient-based methods, we employ a function that acts as an amortised optimiser for the innermost optimisation problem. Interestingly, the amortised optimiser can be viewed as a mover since it strategically shifts around data points. The resulting formulation is a sequential min-max-min game with 3 players: the generator, the critic, and the mover where the new player, the mover, attempts to fool the critic by shifting the data around. Despite involving three players, we demonstrate that our proposed formulation can be trained reasonably effectively via a simple alternative gradient learning strategy. Compared with the existing Lipschitz-constrained formulations of Wasserstein GAN on CIFAR-10, our model yields significantly better diversity scores than weight clipping and comparable performance to gradient penalty method.
BibTeX
@inproceedings{dam2019three,
  author = {Nhan Dam and Quan Hoang and Trung Le and Tu Dinh Nguyen and Hung Bui and Dinh Phung},
  title = {{Three-Player Wasserstein GAN via Amortised Duality}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=s7ARI8fTsYw},
}
Figure