How Well Do WGANs Estimate the Wasserstein Metric?

Anton Mallasto, Guido Montúfar, Augusto GerolinarXiv 2019 · OpenReview

Abstract

Generative modelling is often cast as minimizing a similarity measure between a data distribution and a model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of a potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required to find the optimal potential. In this work, we study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, in particular, the $c$-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the $c$-transform allows for a more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not perform the best in the generative setting.

Introduction

Generative Adversarial Networks (GANs) have revolutionized the field of unsupervised learning by enabling the generation of high-fidelity synthetic data. Among the various formulations, the Wasserstein GAN (WGAN) has gained significant traction due to its theoretical promise of improved training stability and a meaningful loss metric that correlates with image quality. At its core, the WGAN seeks to minimize the Earth Mover’s (Wasserstein-1) distance between the real data distribution and the model distribution.

However, calculating the Wasserstein distance exactly is computationally prohibitive in high dimensions. To make it tractable, researchers rely on the Kantorovich-Rubinstein duality, which requires the discriminator to be a 1-Lipschitz function. This research investigates a fundamental tension in modern WGANs: the methods we use to approximate this metric are often mathematically inaccurate, yet the methods that provide the most accurate mathematical estimates often fail to produce high-quality images.

Methodology: Enforcing the Lipschitz Constraint

The primary challenge in implementing a WGAN is ensuring the discriminator remains within the space of 1-Lipschitz functions. This study compares four distinct approaches to this problem:

  1. Weight Clipping (WC): The original WGAN approach, which forces the weights of the neural network into a fixed box (e.g., $[-0.01, 0.01]$). While simple, it often leads to optimization difficulties and capacity underutilization.
  2. Gradient Penalty (GP): A popular refinement that adds a regularization term to the loss function, penalizing the discriminator if its gradient norm deviates from unity.
  3. $c$-transform: Derived from optimal transport theory, this method uses a mathematical identity to satisfy the duality constraints automatically. By performing a minimization step over the minibatch to find the "conjugate" of the potential function, the Lipschitz constraint is baked into the objective itself.
  4. $(c, \epsilon)$-transform: An entropic relaxation of the $c$-transform. Similar to Sinkhorn distances, this version introduces a smoothing parameter ($\epsilon$) that stabilizes the optimization landscape and improves performance across different architectures.

The Estimation Paradox

The central finding of this research is a striking paradox in generative modeling. When evaluating these methods on standard benchmarks—MNIST, CIFAR-10, and CelebA—a clear divide emerges between estimation accuracy and generative performance.

Mathematical Accuracy

In terms of pure metric estimation, the $c$-transform and its entropic variant are vastly superior. When tasked with estimating the true Wasserstein distance between two known sample sets, the $c$-transform achieved near-zero error. For instance, on the MNIST dataset, the $c$-transform recorded an approximation error of only 0.82. In contrast, standard methods like Weight Clipping and Gradient Penalty were significantly less accurate, with errors of 14.98 and 14.89, respectively. Furthermore, the $c$-transform methods converged to these estimates much faster than their penalty-based counterparts.

Generative Quality

Surprisingly, this mathematical precision does not translate to better images. Despite its inaccurate estimation of the Wasserstein metric, Gradient Penalty (GP) consistently produced the most crisp, diverse, and recognizable images. On the CelebA dataset, GP generated high-quality celebrity faces, whereas the theoretically superior $c$-transform produced blurry, "averaged" blobs. This suggests that the $c$-transform may be suffering from a form of over-smoothing, or that the Wasserstein metric itself, when calculated too accurately, is not the ideal objective for visual fidelity.

Discussion and Insights

The study highlights several key takeaways for the machine learning community:

  • GP as a "Productive Failure": The success of Gradient Penalty suggests that its "errors" might actually serve as beneficial regularization. By failing to perfectly capture the Wasserstein distance, GP may be avoiding the pitfalls of the true metric, such as sensitivity to outliers or a tendency toward mean-seeking behavior.
  • Stability of Entropic Relaxation: The $(c, \epsilon)$-transform proved to be much more robust than the standard $c$-transform. It maintained stability across various architectures (MLPs vs. Convolutional Networks) and batch sizes, confirming that entropic smoothing is a powerful tool for stabilizing optimal transport problems.
  • The Role of the Ground Metric: The blurriness observed in $c$-transform results raises questions about the choice of the ground metric (the underlying distance between individual pixels). If the metric itself encourages averaging, an accurate estimator will naturally produce blurry results.

Conclusion

This work demonstrates that while we have the tools to estimate the Wasserstein metric with high precision via the $c$-transform, doing so may actually hinder generative performance. The "sloppy" approximations provided by Gradient Penalty appear to provide a more effective training signal for producing high-quality synthetic data. Future work must bridge this gap: we need to understand whether we should be seeking better metrics entirely, or if there is a way to harness the accuracy of the $c$-transform without sacrificing the visual clarity of our generative models.

Summary updated

Abstract

Generative modelling is often cast as minimizing a similarity measure between a data distribution and a model distribution. Recently, a popular choice for the similarity measure has been the Wasserstein metric, which can be expressed in the Kantorovich duality formulation as the optimum difference of the expected values of a potential function under the real data distribution and the model hypothesis. In practice, the potential is approximated with a neural network and is called the discriminator. Duality constraints on the function class of the discriminator are enforced approximately, and the expectations are estimated from samples. This gives at least three sources of errors: the approximated discriminator and constraints, the estimation of the expectation value, and the optimization required to find the optimal potential. In this work, we study how well the methods, that are used in generative adversarial networks to approximate the Wasserstein metric, perform. We consider, in particular, the $c$-transform formulation, which eliminates the need to enforce the constraints explicitly. We demonstrate that the $c$-transform allows for a more accurate estimation of the true Wasserstein metric from samples, but surprisingly, does not perform the best in the generative setting.
BibTeX
@misc{mallasto2019how,
  author = {Anton Mallasto and Guido Montúfar and Augusto Gerolin},
  title = {{How Well Do WGANs Estimate the Wasserstein Metric?}},
  year = {2019},
  eprint = {1910.03875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.03875},
}
Figure